{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "g9QvCK6Su4hY"
      },
      "source": [
        "##### Copyright 2018 The AdaNet Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "EUman0Uju6SR"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l5CQ48UOxrqO"
      },
      "source": [
        "# The AdaNet objective"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3tw35taebIOz"
      },
      "source": [
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/adanet/blob/master/adanet/examples/tutorials/adanet_objective.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/adanet/blob/master/adanet/examples/tutorials/adanet_objective.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "7kL896ITmpoR"
      },
      "source": [
        "One of key contributions from *AdaNet: Adaptive Structural Learning of Neural\n",
        "Networks* [[Cortes et al., ICML 2017](https://arxiv.org/abs/1607.01097)] is\n",
        "defining an algorithm that aims to directly minimize the DeepBoost\n",
        "generalization bound from *Deep Boosting*\n",
        "[[Cortes et al., ICML 2014](http://proceedings.mlr.press/v32/cortesb14.pdf)]\n",
        "when applied to neural networks. This algorithm, called **AdaNet**, adaptively\n",
        "grows a neural network as an ensemble of subnetworks that minimizes the AdaNet\n",
        "objective (a.k.a. AdaNet loss):\n",
        "\n",
        "$$F(w) = \\frac{1}{m} \\sum_{i=1}^{m} \\Phi \\left(\\sum_{j=1}^{N}w_jh_j(x_i), y_i \\right) + \\sum_{j=1}^{N} \\left(\\lambda r(h_j) + \\beta \\right) |w_j| $$\n",
        "\n",
        "where $w$ is the set of mixture weights, one per subnetwork $h$,\n",
        "$\\Phi$ is a surrogate loss function such as logistic loss or MSE, $r$ is a\n",
        "function for measuring a subnetwork's complexity, and $\\lambda$ and $\\beta$\n",
        "are hyperparameters.\n",
        "\n",
        "## Mixture weights\n",
        "\n",
        "So what are mixture weights? When forming an ensemble $f$ of subnetworks $h$,\n",
        "we need to somehow combine their predictions. This is done by multiplying\n",
        "the outputs of subnetwork $h_i$ with mixture weight $w_i$, and summing the\n",
        "results:\n",
        "\n",
        "$$f(x) = \\sum_{j=1}^{N}w_jh_j(x)$$\n",
        "\n",
        "In practice, most commonly used set of mixture weight is **uniform average\n",
        "weighting**:\n",
        "\n",
        "$$f(x) = \\frac{1}{N}\\sum_{j=1}^{N}h_j(x)$$\n",
        "\n",
        "However, we can also solve a convex optimization problem to learn the mixture\n",
        "weights that minimize the loss function $\\Phi$:\n",
        "\n",
        "$$F(w) = \\frac{1}{m} \\sum_{i=1}^{m} \\Phi \\left(\\sum_{j=1}^{N}w_jh_j(x_i), y_i \\right)$$\n",
        "\n",
        "This is the first term in the AdaNet objective. The second term applies L1\n",
        "regularization to the mixture weights:\n",
        "\n",
        "$$\\sum_{j=1}^{N} \\left(\\lambda r(h_j) + \\beta \\right) |w_j|$$\n",
        "\n",
        "When $\\lambda \u003e 0$ this penalty serves to prevent the optimization from\n",
        "assigning too much weight to more complex subnetworks according to the\n",
        "complexity measure function $r$.\n",
        "\n",
        "## How AdaNet uses the objective\n",
        "\n",
        "This objective function serves two purposes:\n",
        "\n",
        "1.  To **learn to scale/transform the outputs of each subnetwork $h$** as part\n",
        "    of the ensemble.\n",
        "2.  To **select the best candidate subnetwork $h$** at each AdaNet iteration\n",
        "    to include in the ensemble.\n",
        "\n",
        "Effectively, when learning mixture weights $w$, AdaNet solves a convex\n",
        "combination of the outputs of the frozen subnetworks $h$. For $\\lambda \u003e0$,\n",
        "AdaNet penalizes more complex subnetworks with greater L1 regularization on\n",
        "their mixture weight, and will be less likely to select more complex subnetworks\n",
        "to add to the ensemble at each iteration.\n",
        "\n",
        "In this tutorial, in you will observe the benefits of using AdaNet to learn the\n",
        "ensemble's mixture weights and to perform candidate selection.\n",
        "\n"]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 190
        },
        "colab_type": "code",
        "id": "uWSc6JsqAGjU",
        "outputId": "4b6b3829-496d-482c-ceb1-6f08cf5e5231"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting adanet\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e9/20/57c776bf4574c6e666bfb4434ace74755fbefc33906e23a606b279d4cf81/adanet-0.5.0-py2.py3-none-any.whl (69kB)\n",
            "\u001b[K    100% |████████████████████████████████| 71kB 4.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy\u003e=1.12.0 in /usr/local/lib/python2.7/dist-packages (from adanet) (1.14.6)\n",
            "Requirement already satisfied: six\u003e=1.10.0 in /usr/local/lib/python2.7/dist-packages (from adanet) (1.11.0)\n",
            "Requirement already satisfied: protobuf\u003e=3.6.0 in /usr/local/lib/python2.7/dist-packages (from adanet) (3.6.1)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python2.7/dist-packages (from protobuf\u003e=3.6.0-\u003eadanet) (40.6.2)\n",
            "Installing collected packages: adanet\n",
            "Successfully installed adanet-0.5.0\n"
          ]
        }
      ],
      "source": [
        "#@test {\"skip\": true}\n",
        "# If you're running this in Colab, first install the adanet package:\n",
        "!pip install adanet"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "t4ptB-vwWEGH"
      },
      "outputs": [],
      "source": [
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import functools\n",
        "import os\n",
        "import shutil\n",
        "\n",
        "import adanet\n",
        "import tensorflow as tf\n",
        "\n",
        "# The random seed to use.\n",
        "RANDOM_SEED = 42\n",
        "\n",
        "LOG_DIR = '/tmp/models'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ElcMSpyg_dYO"
      },
      "source": [
        "## Boston Housing dataset\n",
        "\n",
        "In this example, we will solve a regression task known as the [Boston Housing dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html) to predict the price of suburban houses in Boston, MA in the 1970s. There are 13 numerical features, the labels are in thousands of dollars, and there are only 506 examples.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Mp5TTBOJ_ZTU"
      },
      "source": [
        "## Download the data\n",
        "Conveniently, the data is available via Keras:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "colab_type": "code",
        "id": "Plx4LtD4_XFY",
        "outputId": "f99fea93-e6d9-47e3-be7f-23cb19ac40d8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/boston_housing.npz\n",
            "57344/57026 [==============================] - 0s 0us/step\n",
            "65536/57026 [==================================] - 0s 0us/step\n",
            "Model inputs: [  1.23247   0.        8.14      0.        0.538     6.142    91.7\n",
            "   3.9769    4.      307.       21.      396.9      18.72   ] \n",
            "\n",
            "Model output (house price): $15200.0 \n"
          ]
        }
      ],
      "source": [
        "(x_train, y_train), (x_test, y_test) = (\n",
        "    tf.keras.datasets.boston_housing.load_data())\n",
        "\n",
        "# Preview the first example from the training data\n",
        "print('Model inputs: %s \\n' % x_train[0])\n",
        "print('Model output (house price): $%s ' % (y_train[0] * 1000))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bUSb0JtOA5Xc"
      },
      "source": [
        "## Supply the data in TensorFlow\n",
        "\n",
        "Our first task is to supply the data in TensorFlow. Using the\n",
        "tf.estimator.Estimator convention, we will define a function that returns an\n",
        "input_fn which returns feature and label Tensors.\n",
        "\n",
        "We will also use the tf.data.Dataset API to feed the data into our models.\n",
        "\n",
        "Also, as a preprocessing step, we will apply `tf.log1p` to log-scale the\n",
        "features and labels for improved numerical stability during training. To recover\n",
        "the model's predictions in the correct scale, you can apply `tf.math.expm1` to the\n",
        "prediction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "H1U6DppVwTPM"
      },
      "outputs": [],
      "source": [
        "FEATURES_KEY = \"x\"\n",
        "\n",
        "\n",
        "def input_fn(partition, training, batch_size):\n",
        "  \"\"\"Generate an input function for the Estimator.\"\"\"\n",
        "\n",
        "  def _input_fn():\n",
        "\n",
        "    if partition == \"train\":\n",
        "      dataset = tf.data.Dataset.from_tensor_slices(({\n",
        "          FEATURES_KEY: tf.log1p(x_train)\n",
        "      }, tf.log1p(y_train)))\n",
        "    else:\n",
        "      dataset = tf.data.Dataset.from_tensor_slices(({\n",
        "          FEATURES_KEY: tf.log1p(x_test)\n",
        "      }, tf.log1p(y_test)))\n",
        "\n",
        "    # We call repeat after shuffling, rather than before, to prevent separate\n",
        "    # epochs from blending together.\n",
        "    if training:\n",
        "      dataset = dataset.shuffle(10 * batch_size, seed=RANDOM_SEED).repeat()\n",
        "\n",
        "    dataset = dataset.batch(batch_size)\n",
        "    iterator = dataset.make_one_shot_iterator()\n",
        "    features, labels = iterator.get_next()\n",
        "    return features, labels\n",
        "\n",
        "  return _input_fn"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "XXy9pqlq_PQx"
      },
      "source": [
        "## Define the subnetwork generator\n",
        "\n",
        "Let's define a subnetwork generator similar to the one in\n",
        "[[Cortes et al., ICML 2017](https://arxiv.org/abs/1607.01097)] and in\n",
        "`simple_dnn.py` which creates two candidate fully-connected neural networks at\n",
        "each iteration with the same width, but one an additional hidden layer. To make\n",
        "our generator *adaptive*, each subnetwork will have at least the same number\n",
        "of hidden layers as the most recently added subnetwork to the\n",
        "`previous_ensemble`.\n",
        "\n",
        "We define the complexity measure function $r$ to be $r(h) = \\sqrt{d(h)}$, where\n",
        "$d$ is the number of hidden layers in the neural network $h$, to approximate the\n",
        "Rademacher bounds from\n",
        "[[Golowich et. al, 2017](https://arxiv.org/abs/1712.06541)]. So subnetworks\n",
        "with more hidden layers, and therefore more capacity, will have more heavily\n",
        "regularized mixture weights."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "hZn9VeFUHvD-"
      },
      "outputs": [],
      "source": [
        "_NUM_LAYERS_KEY = \"num_layers\"\n",
        "\n",
        "\n",
        "class _SimpleDNNBuilder(adanet.subnetwork.Builder):\n",
        "  \"\"\"Builds a DNN subnetwork for AdaNet.\"\"\"\n",
        "\n",
        "  def __init__(self, optimizer, layer_size, num_layers, learn_mixture_weights,\n",
        "               seed):\n",
        "    \"\"\"Initializes a `_DNNBuilder`.\n",
        "\n",
        "    Args:\n",
        "      optimizer: An `Optimizer` instance for training both the subnetwork and\n",
        "        the mixture weights.\n",
        "      layer_size: The number of nodes to output at each hidden layer.\n",
        "      num_layers: The number of hidden layers.\n",
        "      learn_mixture_weights: Whether to solve a learning problem to find the\n",
        "        best mixture weights, or use their default value according to the\n",
        "        mixture weight type. When `False`, the subnetworks will return a no_op\n",
        "        for the mixture weight train op.\n",
        "      seed: A random seed.\n",
        "\n",
        "    Returns:\n",
        "      An instance of `_SimpleDNNBuilder`.\n",
        "    \"\"\"\n",
        "\n",
        "    self._optimizer = optimizer\n",
        "    self._layer_size = layer_size\n",
        "    self._num_layers = num_layers\n",
        "    self._learn_mixture_weights = learn_mixture_weights\n",
        "    self._seed = seed\n",
        "\n",
        "  def build_subnetwork(self,\n",
        "                       features,\n",
        "                       logits_dimension,\n",
        "                       training,\n",
        "                       iteration_step,\n",
        "                       summary,\n",
        "                       previous_ensemble=None):\n",
        "    \"\"\"See `adanet.subnetwork.Builder`.\"\"\"\n",
        "\n",
        "    input_layer = tf.to_float(features[FEATURES_KEY])\n",
        "    kernel_initializer = tf.glorot_uniform_initializer(seed=self._seed)\n",
        "    last_layer = input_layer\n",
        "    for _ in range(self._num_layers):\n",
        "      last_layer = tf.layers.dense(\n",
        "          last_layer,\n",
        "          units=self._layer_size,\n",
        "          activation=tf.nn.relu,\n",
        "          kernel_initializer=kernel_initializer)\n",
        "    logits = tf.layers.dense(\n",
        "        last_layer,\n",
        "        units=logits_dimension,\n",
        "        kernel_initializer=kernel_initializer)\n",
        "\n",
        "    persisted_tensors = {_NUM_LAYERS_KEY: tf.constant(self._num_layers)}\n",
        "    return adanet.Subnetwork(\n",
        "        last_layer=last_layer,\n",
        "        logits=logits,\n",
        "        complexity=self._measure_complexity(),\n",
        "        persisted_tensors=persisted_tensors)\n",
        "\n",
        "  def _measure_complexity(self):\n",
        "    \"\"\"Approximates Rademacher complexity as the square-root of the depth.\"\"\"\n",
        "    return tf.sqrt(tf.to_float(self._num_layers))\n",
        "\n",
        "  def build_subnetwork_train_op(self, subnetwork, loss, var_list, labels,\n",
        "                                iteration_step, summary, previous_ensemble):\n",
        "    \"\"\"See `adanet.subnetwork.Builder`.\"\"\"\n",
        "    return self._optimizer.minimize(loss=loss, var_list=var_list)\n",
        "\n",
        "  def build_mixture_weights_train_op(self, loss, var_list, logits, labels,\n",
        "                                     iteration_step, summary):\n",
        "    \"\"\"See `adanet.subnetwork.Builder`.\"\"\"\n",
        "\n",
        "    if not self._learn_mixture_weights:\n",
        "      return tf.no_op()\n",
        "    return self._optimizer.minimize(loss=loss, var_list=var_list)\n",
        "\n",
        "  @property\n",
        "  def name(self):\n",
        "    \"\"\"See `adanet.subnetwork.Builder`.\"\"\"\n",
        "\n",
        "    if self._num_layers == 0:\n",
        "      # A DNN with no hidden layers is a linear model.\n",
        "      return \"linear\"\n",
        "    return \"{}_layer_dnn\".format(self._num_layers)\n",
        "\n",
        "\n",
        "class SimpleDNNGenerator(adanet.subnetwork.Generator):\n",
        "  \"\"\"Generates a two DNN subnetworks at each iteration.\n",
        "\n",
        "  The first DNN has an identical shape to the most recently added subnetwork\n",
        "  in `previous_ensemble`. The second has the same shape plus one more dense\n",
        "  layer on top. This is similar to the adaptive network presented in Figure 2 of\n",
        "  [Cortes et al. ICML 2017](https://arxiv.org/abs/1607.01097), without the\n",
        "  connections to hidden layers of networks from previous iterations.\n",
        "  \"\"\"\n",
        "\n",
        "  def __init__(self,\n",
        "               optimizer,\n",
        "               layer_size=64,\n",
        "               learn_mixture_weights=False,\n",
        "               seed=None):\n",
        "    \"\"\"Initializes a DNN `Generator`.\n",
        "\n",
        "    Args:\n",
        "      optimizer: An `Optimizer` instance for training both the subnetwork and\n",
        "        the mixture weights.\n",
        "      layer_size: Number of nodes in each hidden layer of the subnetwork\n",
        "        candidates. Note that this parameter is ignored in a DNN with no hidden\n",
        "        layers.\n",
        "      learn_mixture_weights: Whether to solve a learning problem to find the\n",
        "        best mixture weights, or use their default value according to the\n",
        "        mixture weight type. When `False`, the subnetworks will return a no_op\n",
        "        for the mixture weight train op.\n",
        "      seed: A random seed.\n",
        "\n",
        "    Returns:\n",
        "      An instance of `Generator`.\n",
        "    \"\"\"\n",
        "\n",
        "    self._seed = seed\n",
        "    self._dnn_builder_fn = functools.partial(\n",
        "        _SimpleDNNBuilder,\n",
        "        optimizer=optimizer,\n",
        "        layer_size=layer_size,\n",
        "        learn_mixture_weights=learn_mixture_weights)\n",
        "\n",
        "  def generate_candidates(self, previous_ensemble, iteration_number,\n",
        "                          previous_ensemble_reports, all_reports):\n",
        "    \"\"\"See `adanet.subnetwork.Generator`.\"\"\"\n",
        "\n",
        "    num_layers = 0\n",
        "    seed = self._seed\n",
        "    if previous_ensemble:\n",
        "      num_layers = tf.contrib.util.constant_value(\n",
        "          previous_ensemble.weighted_subnetworks[\n",
        "              -1].subnetwork.persisted_tensors[_NUM_LAYERS_KEY])\n",
        "    if seed is not None:\n",
        "      seed += iteration_number\n",
        "    return [\n",
        "        self._dnn_builder_fn(num_layers=num_layers, seed=seed),\n",
        "        self._dnn_builder_fn(num_layers=num_layers + 1, seed=seed),\n",
        "    ]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fhu2zpf9faIB"
      },
      "source": [
        "## Launch TensorBoard\n",
        "\n",
        "Let's run [TensorBoard](https://www.tensorflow.org/guide/summaries_and_tensorboard) to visualize model training over time. We'll use [ngrok](https://ngrok.com/) to tunnel traffic to localhost.\n",
        "\n",
        "*The instructions for setting up Tensorboard were obtained from https://www.dlology.com/blog/quick-guide-to-run-tensorboard-in-google-colab/*\n",
        "\n",
        "Run the next cells and follow the link to see the TensorBoard in a new tab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "MLo_xWQMe686"
      },
      "outputs": [],
      "source": [
        "#@test {\"skip\": true}\n",
        "\n",
        "get_ipython().system_raw(\n",
        "    'tensorboard --logdir {} --host 0.0.0.0 --port 6006 \u0026'\n",
        "    .format(LOG_DIR)\n",
        ")\n",
        "\n",
        "# Install ngrok binary.\n",
        "! wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n",
        "! unzip ngrok-stable-linux-amd64.zip\n",
        "\n",
        "# Delete old logs dir.\n",
        "shutil.rmtree(LOG_DIR, ignore_errors=True)\n",
        "\n",
        "print(\"Follow this link to open TensorBoard in a new tab.\")\n",
        "get_ipython().system_raw('./ngrok http 6006 \u0026')\n",
        "! curl -s http://localhost:4040/api/tunnels | python3 -c \\\n",
        "    \"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])\"\n",
        "\n"]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "vJRH-v5NhnY9"
      },
      "source": [
        "## Train and evaluate\n",
        "\n",
        "Next we create an `adanet.Estimator` using the `SimpleDNNGenerator` we just defined.\n",
        "\n",
        "In this section we will show the effects of two hyperparamters: **learning mixture weights** and **complexity regularization**.\n",
        "\n",
        "On the righthand side you will be able to play with the hyperparameters of this model. Until you reach the end of this section, we ask that you not change them. \n",
        "\n",
        "At first we will not learn the mixture weights, using their default initial value. Here they will be scalars initialized to $1/N$ where $N$ is the number of subnetworks in the ensemble, effectively creating a **uniform average ensemble**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "8TJHr1k-zYpN"
      },
      "outputs": [],
      "source": [
        "#@title AdaNet parameters\n",
        "LEARNING_RATE = 0.001  #@param {type:\"number\"}\n",
        "TRAIN_STEPS = 60000  #@param {type:\"integer\"}\n",
        "BATCH_SIZE = 32  #@param {type:\"integer\"}\n",
        "\n",
        "LEARN_MIXTURE_WEIGHTS = False  #@param {type:\"boolean\"}\n",
        "ADANET_LAMBDA = 0  #@param {type:\"number\"}\n",
        "ADANET_ITERATIONS = 3  #@param {type:\"integer\"}\n",
        "\n",
        "\n",
        "def train_and_evaluate(experiment_name, learn_mixture_weights=LEARN_MIXTURE_WEIGHTS,\n",
        "                       adanet_lambda=ADANET_LAMBDA):\n",
        "  \"\"\"Trains an `adanet.Estimator` to predict housing prices.\"\"\"\n",
        "\n",
        "  model_dir = os.path.join(LOG_DIR, experiment_name)\n",
        "\n",
        "  estimator = adanet.Estimator(\n",
        "      # Since we are predicting housing prices, we'll use a regression\n",
        "      # head that optimizes for MSE.\n",
        "      head=tf.contrib.estimator.regression_head(\n",
        "          loss_reduction=tf.losses.Reduction.SUM_OVER_BATCH_SIZE),\n",
        "\n",
        "      # Define the generator, which defines our search space of subnetworks\n",
        "      # to train as candidates to add to the final AdaNet model.\n",
        "      subnetwork_generator=SimpleDNNGenerator(\n",
        "          optimizer=tf.train.RMSPropOptimizer(learning_rate=LEARNING_RATE),\n",
        "          learn_mixture_weights=learn_mixture_weights,\n",
        "          seed=RANDOM_SEED),\n",
        "\n",
        "      # Lambda is a the strength of complexity regularization. A larger\n",
        "      # value will penalize more complex subnetworks.\n",
        "      adanet_lambda=adanet_lambda,\n",
        "\n",
        "      # The number of train steps per iteration.\n",
        "      max_iteration_steps=TRAIN_STEPS // ADANET_ITERATIONS,\n",
        "\n",
        "      # The evaluator will evaluate the model on the full training set to\n",
        "      # compute the overall AdaNet loss (train loss + complexity\n",
        "      # regularization) to select the best candidate to include in the\n",
        "      # final AdaNet model.\n",
        "      evaluator=adanet.Evaluator(\n",
        "          input_fn=input_fn(\"train\", training=False, batch_size=BATCH_SIZE)),\n",
        "\n",
        "      # Configuration for Estimators.\n",
        "      config=tf.estimator.RunConfig(\n",
        "          save_summary_steps=5000,\n",
        "          save_checkpoints_steps=5000,\n",
        "          tf_random_seed=RANDOM_SEED,\n",
        "          model_dir=model_dir))\n",
        "\n",
        "  # Train and evaluate using using the tf.estimator tooling.\n",
        "  train_spec = tf.estimator.TrainSpec(\n",
        "      input_fn=input_fn(\"train\", training=True, batch_size=BATCH_SIZE),\n",
        "      max_steps=TRAIN_STEPS)\n",
        "  eval_spec = tf.estimator.EvalSpec(\n",
        "      input_fn=input_fn(\"test\", training=False, batch_size=BATCH_SIZE),\n",
        "      steps=None,\n",
        "      start_delay_secs=1,\n",
        "      throttle_secs=30,\n",
        "  )\n",
        "  return tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)\n",
        "\n",
        "\n",
        "def ensemble_architecture(result):\n",
        "  \"\"\"Extracts the ensemble architecture from evaluation results.\"\"\"\n",
        "\n",
        "  architecture = result[\"architecture/adanet/ensembles\"]\n",
        "  # The architecture is a serialized Summary proto for TensorBoard.\n",
        "  summary_proto = tf.summary.Summary.FromString(architecture)\n",
        "  return summary_proto.value[0].tensor.string_val[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 29209
        },
        "colab_type": "code",
        "id": "U27fd6Kwhk3r",
        "outputId": "f917b167-6500-4a00-bdfc-bbb366280aac"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': \u003ctensorflow.python.training.server_lib.ClusterSpec object at 0x7f639a17d810\u003e, '_model_dir': '/tmp/models/uniform_average_ensemble_baseline', '_protocol': None, '_save_checkpoints_steps': 5000, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': 42, '_save_summary_steps': 5000, '_device_fn': None, '_experimental_distribute': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_evaluation_master': '', '_eval_distribute': None, '_global_id_in_cluster': 0, '_master': ''}\n",
            "INFO:tensorflow:Not using Distribute Coordinator.\n",
            "INFO:tensorflow:Running training and evaluation locally (non-distributed).\n",
            "INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 5000 or save_checkpoints_secs None.\n",
            "INFO:tensorflow:Beginning training AdaNet iteration 0\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "WARNING:tensorflow:From \u003cipython-input-6-fb8e373057d9\u003e:60: calling __new__ (from adanet.core.subnetwork.generator) with persisted_tensors is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "`persisted_tensors` is deprecated, please use `shared` instead.\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Create CheckpointSaverHook.\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving checkpoints for 0 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:loss = 5.1627307, step = 1\n",
            "INFO:tensorflow:global_step/sec: 281.388\n",
            "INFO:tensorflow:loss = 0.36111063, step = 101 (0.356 sec)\n",
            "INFO:tensorflow:global_step/sec: 632.512\n",
            "INFO:tensorflow:loss = 0.23443298, step = 201 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.624\n",
            "INFO:tensorflow:loss = 0.05358686, step = 301 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 634.308\n",
            "INFO:tensorflow:loss = 0.056381542, step = 401 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.218\n",
            "INFO:tensorflow:loss = 0.06720045, step = 501 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 629.826\n",
            "INFO:tensorflow:loss = 0.0430552, step = 601 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 626.814\n",
            "INFO:tensorflow:loss = 0.02634022, step = 701 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 633.646\n",
            "INFO:tensorflow:loss = 0.02167327, step = 801 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 639.223\n",
            "INFO:tensorflow:loss = 0.03461258, step = 901 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 632.959\n",
            "INFO:tensorflow:loss = 0.10106905, step = 1001 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 584.874\n",
            "INFO:tensorflow:loss = 0.035603803, step = 1101 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.681\n",
            "INFO:tensorflow:loss = 0.05181156, step = 1201 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 636.704\n",
            "INFO:tensorflow:loss = 0.103541166, step = 1301 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 640.824\n",
            "INFO:tensorflow:loss = 0.025568489, step = 1401 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 646.132\n",
            "INFO:tensorflow:loss = 0.017370185, step = 1501 (0.154 sec)\n",
            "INFO:tensorflow:global_step/sec: 640.73\n",
            "INFO:tensorflow:loss = 0.046300475, step = 1601 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 654.467\n",
            "INFO:tensorflow:loss = 0.04478077, step = 1701 (0.153 sec)\n",
            "INFO:tensorflow:global_step/sec: 631.445\n",
            "INFO:tensorflow:loss = 0.037187986, step = 1801 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 575.729\n",
            "INFO:tensorflow:loss = 0.041324023, step = 1901 (0.174 sec)\n",
            "INFO:tensorflow:global_step/sec: 630.163\n",
            "INFO:tensorflow:loss = 0.03305667, step = 2001 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 641.272\n",
            "INFO:tensorflow:loss = 0.047040317, step = 2101 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.523\n",
            "INFO:tensorflow:loss = 0.02897141, step = 2201 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 615.858\n",
            "INFO:tensorflow:loss = 0.034482427, step = 2301 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 631.804\n",
            "INFO:tensorflow:loss = 0.02347619, step = 2401 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 627.955\n",
            "INFO:tensorflow:loss = 0.07312092, step = 2501 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 650.335\n",
            "INFO:tensorflow:loss = 0.037120573, step = 2601 (0.154 sec)\n",
            "INFO:tensorflow:global_step/sec: 548.438\n",
            "INFO:tensorflow:loss = 0.05865065, step = 2701 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 620.456\n",
            "INFO:tensorflow:loss = 0.013324069, step = 2801 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 639.537\n",
            "INFO:tensorflow:loss = 0.03051297, step = 2901 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 630.354\n",
            "INFO:tensorflow:loss = 0.038826123, step = 3001 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 639.223\n",
            "INFO:tensorflow:loss = 0.028354896, step = 3101 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 631.018\n",
            "INFO:tensorflow:loss = 0.02475904, step = 3201 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 501.975\n",
            "INFO:tensorflow:loss = 0.046555758, step = 3301 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 602.21\n",
            "INFO:tensorflow:loss = 0.03615782, step = 3401 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 601.913\n",
            "INFO:tensorflow:loss = 0.05879674, step = 3501 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 609.277\n",
            "INFO:tensorflow:loss = 0.02304113, step = 3601 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 644.118\n",
            "INFO:tensorflow:loss = 0.015732538, step = 3701 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 637.804\n",
            "INFO:tensorflow:loss = 0.02152918, step = 3801 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 653.219\n",
            "INFO:tensorflow:loss = 0.019783746, step = 3901 (0.153 sec)\n",
            "INFO:tensorflow:global_step/sec: 642.884\n",
            "INFO:tensorflow:loss = 0.040957503, step = 4001 (0.155 sec)\n",
            "INFO:tensorflow:global_step/sec: 641.688\n",
            "INFO:tensorflow:loss = 0.01987027, step = 4101 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 607.855\n",
            "INFO:tensorflow:loss = 0.025079198, step = 4201 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 562.6\n",
            "INFO:tensorflow:loss = 0.012929243, step = 4301 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 627.692\n",
            "INFO:tensorflow:loss = 0.045354083, step = 4401 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 644.982\n",
            "INFO:tensorflow:loss = 0.030239534, step = 4501 (0.155 sec)\n",
            "INFO:tensorflow:global_step/sec: 639.26\n",
            "INFO:tensorflow:loss = 0.034541786, step = 4601 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 642.174\n",
            "INFO:tensorflow:loss = 0.04034666, step = 4701 (0.155 sec)\n",
            "INFO:tensorflow:global_step/sec: 638.08\n",
            "INFO:tensorflow:loss = 0.04709795, step = 4801 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.786\n",
            "INFO:tensorflow:loss = 0.021513231, step = 4901 (0.161 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 5000 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-15:57:04\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/model.ckpt-5000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_linear' dict for global step 5000: architecture/adanet/ensembles = \n",
            "W\n",
            "9adanet/iteration_0/ensemble_t0_linear/architecture/adanetB\u0010\u0008\u0007\u0012\u0000B\n",
            "| linear |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.04963816, average_loss/adanet/subnetwork = 0.04963816, average_loss/adanet/uniform_average_ensemble = 0.04963816, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.06355202, loss/adanet/subnetwork = 0.06355202, loss/adanet/uniform_average_ensemble = 0.06355202, prediction/mean/adanet/adanet_weighted_ensemble = 3.1209233, prediction/mean/adanet/subnetwork = 3.1209233, prediction/mean/adanet/uniform_average_ensemble = 3.1209233\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 5000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.04984243, average_loss/adanet/subnetwork = 0.04984243, average_loss/adanet/uniform_average_ensemble = 0.04984243, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.06829582, loss/adanet/subnetwork = 0.06829582, loss/adanet/uniform_average_ensemble = 0.06829582, prediction/mean/adanet/adanet_weighted_ensemble = 3.131058, prediction/mean/adanet/subnetwork = 3.131058, prediction/mean/adanet/uniform_average_ensemble = 3.131058\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-15:57:06\n",
            "INFO:tensorflow:Saving dict for global step 5000: average_loss = 0.04984243, average_loss/adanet/adanet_weighted_ensemble = 0.04984243, average_loss/adanet/subnetwork = 0.04984243, average_loss/adanet/uniform_average_ensemble = 0.04984243, global_step = 5000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.06829582, loss/adanet/adanet_weighted_ensemble = 0.06829582, loss/adanet/subnetwork = 0.06829582, loss/adanet/uniform_average_ensemble = 0.06829582, prediction/mean = 3.131058, prediction/mean/adanet/adanet_weighted_ensemble = 3.131058, prediction/mean/adanet/subnetwork = 3.131058, prediction/mean/adanet/uniform_average_ensemble = 3.131058\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmp/models/uniform_average_ensemble_baseline/model.ckpt-5000\n",
            "INFO:tensorflow:global_step/sec: 27.6979\n",
            "INFO:tensorflow:loss = 0.0135752205, step = 5001 (3.614 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.228\n",
            "INFO:tensorflow:loss = 0.019322123, step = 5101 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.517\n",
            "INFO:tensorflow:loss = 0.038781, step = 5201 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 448.632\n",
            "INFO:tensorflow:loss = 0.035238247, step = 5301 (0.223 sec)\n",
            "INFO:tensorflow:global_step/sec: 641.725\n",
            "INFO:tensorflow:loss = 0.022843715, step = 5401 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 634.2\n",
            "INFO:tensorflow:loss = 0.017707743, step = 5501 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 658.142\n",
            "INFO:tensorflow:loss = 0.02330019, step = 5601 (0.152 sec)\n",
            "INFO:tensorflow:global_step/sec: 642.026\n",
            "INFO:tensorflow:loss = 0.019027822, step = 5701 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 654.368\n",
            "INFO:tensorflow:loss = 0.020987503, step = 5801 (0.153 sec)\n",
            "INFO:tensorflow:global_step/sec: 659.362\n",
            "INFO:tensorflow:loss = 0.028692458, step = 5901 (0.151 sec)\n",
            "INFO:tensorflow:global_step/sec: 650.326\n",
            "INFO:tensorflow:loss = 0.017491236, step = 6001 (0.154 sec)\n",
            "INFO:tensorflow:global_step/sec: 647.102\n",
            "INFO:tensorflow:loss = 0.03423073, step = 6101 (0.155 sec)\n",
            "INFO:tensorflow:global_step/sec: 650.039\n",
            "INFO:tensorflow:loss = 0.043175362, step = 6201 (0.154 sec)\n",
            "INFO:tensorflow:global_step/sec: 633.156\n",
            "INFO:tensorflow:loss = 0.024462596, step = 6301 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 601.684\n",
            "INFO:tensorflow:loss = 0.034094155, step = 6401 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 656.134\n",
            "INFO:tensorflow:loss = 0.03336587, step = 6501 (0.152 sec)\n",
            "INFO:tensorflow:global_step/sec: 630.283\n",
            "INFO:tensorflow:loss = 0.03862817, step = 6601 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.204\n",
            "INFO:tensorflow:loss = 0.02139195, step = 6701 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 643.318\n",
            "INFO:tensorflow:loss = 0.027588185, step = 6801 (0.155 sec)\n",
            "INFO:tensorflow:global_step/sec: 651.479\n",
            "INFO:tensorflow:loss = 0.010055454, step = 6901 (0.154 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.511\n",
            "INFO:tensorflow:loss = 0.019065816, step = 7001 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 652.631\n",
            "INFO:tensorflow:loss = 0.035144374, step = 7101 (0.154 sec)\n",
            "INFO:tensorflow:global_step/sec: 655.214\n",
            "INFO:tensorflow:loss = 0.015206728, step = 7201 (0.152 sec)\n",
            "INFO:tensorflow:global_step/sec: 654.214\n",
            "INFO:tensorflow:loss = 0.041717835, step = 7301 (0.153 sec)\n",
            "INFO:tensorflow:global_step/sec: 652.623\n",
            "INFO:tensorflow:loss = 0.02104327, step = 7401 (0.153 sec)\n",
            "INFO:tensorflow:global_step/sec: 627.676\n",
            "INFO:tensorflow:loss = 0.016285382, step = 7501 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 637.089\n",
            "INFO:tensorflow:loss = 0.012007719, step = 7601 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 650.702\n",
            "INFO:tensorflow:loss = 0.024921771, step = 7701 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 625.336\n",
            "INFO:tensorflow:loss = 0.018093377, step = 7801 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 634.792\n",
            "INFO:tensorflow:loss = 0.013643123, step = 7901 (0.155 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.761\n",
            "INFO:tensorflow:loss = 0.025414744, step = 8001 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 644.446\n",
            "INFO:tensorflow:loss = 0.029413532, step = 8101 (0.155 sec)\n",
            "INFO:tensorflow:global_step/sec: 626.543\n",
            "INFO:tensorflow:loss = 0.0742735, step = 8201 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.846\n",
            "INFO:tensorflow:loss = 0.021159641, step = 8301 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 625.234\n",
            "INFO:tensorflow:loss = 0.04715711, step = 8401 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 574.845\n",
            "INFO:tensorflow:loss = 0.01712554, step = 8501 (0.174 sec)\n",
            "INFO:tensorflow:global_step/sec: 636.704\n",
            "INFO:tensorflow:loss = 0.025253613, step = 8601 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 642.583\n",
            "INFO:tensorflow:loss = 0.021093639, step = 8701 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 597.236\n",
            "INFO:tensorflow:loss = 0.047748983, step = 8801 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 638.929\n",
            "INFO:tensorflow:loss = 0.042816155, step = 8901 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 646.805\n",
            "INFO:tensorflow:loss = 0.04366387, step = 9001 (0.155 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.169\n",
            "INFO:tensorflow:loss = 0.019078696, step = 9101 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 637.658\n",
            "INFO:tensorflow:loss = 0.030207923, step = 9201 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 635.283\n",
            "INFO:tensorflow:loss = 0.012874163, step = 9301 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 641.614\n",
            "INFO:tensorflow:loss = 0.012613616, step = 9401 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 638.202\n",
            "INFO:tensorflow:loss = 0.050132595, step = 9501 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 620.144\n",
            "INFO:tensorflow:loss = 0.022509016, step = 9601 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 635.373\n",
            "INFO:tensorflow:loss = 0.009314457, step = 9701 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 643.836\n",
            "INFO:tensorflow:loss = 0.017946877, step = 9801 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 611.513\n",
            "INFO:tensorflow:loss = 0.019298766, step = 9901 (0.164 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 10000 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-15:57:16\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/model.ckpt-10000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_linear' dict for global step 10000: architecture/adanet/ensembles = \n",
            "W\n",
            "9adanet/iteration_0/ensemble_t0_linear/architecture/adanetB\u0010\u0008\u0007\u0012\u0000B\n",
            "| linear |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.050082315, average_loss/adanet/subnetwork = 0.050082315, average_loss/adanet/uniform_average_ensemble = 0.050082315, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.06438362, loss/adanet/subnetwork = 0.06438362, loss/adanet/uniform_average_ensemble = 0.06438362, prediction/mean/adanet/adanet_weighted_ensemble = 3.118966, prediction/mean/adanet/subnetwork = 3.118966, prediction/mean/adanet/uniform_average_ensemble = 3.118966\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 10000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.045428075, average_loss/adanet/subnetwork = 0.045428075, average_loss/adanet/uniform_average_ensemble = 0.045428075, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.054026034, loss/adanet/subnetwork = 0.054026034, loss/adanet/uniform_average_ensemble = 0.054026034, prediction/mean/adanet/adanet_weighted_ensemble = 3.069825, prediction/mean/adanet/subnetwork = 3.069825, prediction/mean/adanet/uniform_average_ensemble = 3.069825\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-15:57:16\n",
            "INFO:tensorflow:Saving dict for global step 10000: average_loss = 0.045428075, average_loss/adanet/adanet_weighted_ensemble = 0.045428075, average_loss/adanet/subnetwork = 0.045428075, average_loss/adanet/uniform_average_ensemble = 0.045428075, global_step = 10000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.054026034, loss/adanet/adanet_weighted_ensemble = 0.054026034, loss/adanet/subnetwork = 0.054026034, loss/adanet/uniform_average_ensemble = 0.054026034, prediction/mean = 3.069825, prediction/mean/adanet/adanet_weighted_ensemble = 3.069825, prediction/mean/adanet/subnetwork = 3.069825, prediction/mean/adanet/uniform_average_ensemble = 3.069825\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 10000: /tmp/models/uniform_average_ensemble_baseline/model.ckpt-10000\n",
            "INFO:tensorflow:global_step/sec: 35.2907\n",
            "INFO:tensorflow:loss = 0.021735623, step = 10001 (2.835 sec)\n",
            "INFO:tensorflow:global_step/sec: 630.776\n",
            "INFO:tensorflow:loss = 0.022340164, step = 10101 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 640.414\n",
            "INFO:tensorflow:loss = 0.031335674, step = 10201 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 634.26\n",
            "INFO:tensorflow:loss = 0.02913072, step = 10301 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 640.185\n",
            "INFO:tensorflow:loss = 0.022426063, step = 10401 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 627.676\n",
            "INFO:tensorflow:loss = 0.03042262, step = 10501 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.505\n",
            "INFO:tensorflow:loss = 0.013473484, step = 10601 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 630.879\n",
            "INFO:tensorflow:loss = 0.00851935, step = 10701 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.138\n",
            "INFO:tensorflow:loss = 0.016731912, step = 10801 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 637.666\n",
            "INFO:tensorflow:loss = 0.02075386, step = 10901 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 646.521\n",
            "INFO:tensorflow:loss = 0.0093052, step = 11001 (0.155 sec)\n",
            "INFO:tensorflow:global_step/sec: 639.141\n",
            "INFO:tensorflow:loss = 0.0222205, step = 11101 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 632.295\n",
            "INFO:tensorflow:loss = 0.009712502, step = 11201 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 649.743\n",
            "INFO:tensorflow:loss = 0.025909176, step = 11301 (0.154 sec)\n",
            "INFO:tensorflow:global_step/sec: 655.626\n",
            "INFO:tensorflow:loss = 0.015414276, step = 11401 (0.152 sec)\n",
            "INFO:tensorflow:global_step/sec: 640.48\n",
            "INFO:tensorflow:loss = 0.05516013, step = 11501 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 635.473\n",
            "INFO:tensorflow:loss = 0.0075291377, step = 11601 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 633.968\n",
            "INFO:tensorflow:loss = 0.031882644, step = 11701 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 651.257\n",
            "INFO:tensorflow:loss = 0.013129325, step = 11801 (0.152 sec)\n",
            "INFO:tensorflow:global_step/sec: 636.631\n",
            "INFO:tensorflow:loss = 0.033133373, step = 11901 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 640.413\n",
            "INFO:tensorflow:loss = 0.02534269, step = 12001 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 653.676\n",
            "INFO:tensorflow:loss = 0.015705975, step = 12101 (0.153 sec)\n",
            "INFO:tensorflow:global_step/sec: 634.691\n",
            "INFO:tensorflow:loss = 0.010178973, step = 12201 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 645.725\n",
            "INFO:tensorflow:loss = 0.015422712, step = 12301 (0.155 sec)\n",
            "INFO:tensorflow:global_step/sec: 644.658\n",
            "INFO:tensorflow:loss = 0.0123514235, step = 12401 (0.155 sec)\n",
            "INFO:tensorflow:global_step/sec: 640.918\n",
            "INFO:tensorflow:loss = 0.018206993, step = 12501 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 642.901\n",
            "INFO:tensorflow:loss = 0.018180907, step = 12601 (0.155 sec)\n",
            "INFO:tensorflow:global_step/sec: 628.133\n",
            "INFO:tensorflow:loss = 0.02052683, step = 12701 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.01\n",
            "INFO:tensorflow:loss = 0.016060527, step = 12801 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 638.215\n",
            "INFO:tensorflow:loss = 0.014817359, step = 12901 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 632.554\n",
            "INFO:tensorflow:loss = 0.02727741, step = 13001 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 638.614\n",
            "INFO:tensorflow:loss = 0.03937371, step = 13101 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.449\n",
            "INFO:tensorflow:loss = 0.021417897, step = 13201 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 634.401\n",
            "INFO:tensorflow:loss = 0.010210026, step = 13301 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 633.641\n",
            "INFO:tensorflow:loss = 0.017274674, step = 13401 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 628.157\n",
            "INFO:tensorflow:loss = 0.019349674, step = 13501 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 631.78\n",
            "INFO:tensorflow:loss = 0.011177426, step = 13601 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 593.422\n",
            "INFO:tensorflow:loss = 0.006929117, step = 13701 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 645.469\n",
            "INFO:tensorflow:loss = 0.03895629, step = 13801 (0.155 sec)\n",
            "INFO:tensorflow:global_step/sec: 636.181\n",
            "INFO:tensorflow:loss = 0.014666822, step = 13901 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 651.665\n",
            "INFO:tensorflow:loss = 0.016907558, step = 14001 (0.153 sec)\n",
            "INFO:tensorflow:global_step/sec: 642.327\n",
            "INFO:tensorflow:loss = 0.015120897, step = 14101 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 628.259\n",
            "INFO:tensorflow:loss = 0.014633164, step = 14201 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 639.509\n",
            "INFO:tensorflow:loss = 0.0136266565, step = 14301 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 630.37\n",
            "INFO:tensorflow:loss = 0.008968145, step = 14401 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.882\n",
            "INFO:tensorflow:loss = 0.013421325, step = 14501 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 636.931\n",
            "INFO:tensorflow:loss = 0.029463077, step = 14601 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 638.757\n",
            "INFO:tensorflow:loss = 0.018902685, step = 14701 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 620.278\n",
            "INFO:tensorflow:loss = 0.027415033, step = 14801 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 627.168\n",
            "INFO:tensorflow:loss = 0.031739023, step = 14901 (0.159 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 15000 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-15:57:26\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/model.ckpt-15000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_linear' dict for global step 15000: architecture/adanet/ensembles = \n",
            "W\n",
            "9adanet/iteration_0/ensemble_t0_linear/architecture/adanetB\u0010\u0008\u0007\u0012\u0000B\n",
            "| linear |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.05043939, average_loss/adanet/subnetwork = 0.05043939, average_loss/adanet/uniform_average_ensemble = 0.05043939, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.06300935, loss/adanet/subnetwork = 0.06300935, loss/adanet/uniform_average_ensemble = 0.06300935, prediction/mean/adanet/adanet_weighted_ensemble = 3.0819287, prediction/mean/adanet/subnetwork = 3.0819287, prediction/mean/adanet/uniform_average_ensemble = 3.0819287\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 15000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.049884323, average_loss/adanet/subnetwork = 0.049884323, average_loss/adanet/uniform_average_ensemble = 0.049884323, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.048252657, loss/adanet/subnetwork = 0.048252657, loss/adanet/uniform_average_ensemble = 0.048252657, prediction/mean/adanet/adanet_weighted_ensemble = 2.9956787, prediction/mean/adanet/subnetwork = 2.9956787, prediction/mean/adanet/uniform_average_ensemble = 2.9956787\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-15:57:27\n",
            "INFO:tensorflow:Saving dict for global step 15000: average_loss = 0.049884323, average_loss/adanet/adanet_weighted_ensemble = 0.049884323, average_loss/adanet/subnetwork = 0.049884323, average_loss/adanet/uniform_average_ensemble = 0.049884323, global_step = 15000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.048252657, loss/adanet/adanet_weighted_ensemble = 0.048252657, loss/adanet/subnetwork = 0.048252657, loss/adanet/uniform_average_ensemble = 0.048252657, prediction/mean = 2.9956787, prediction/mean/adanet/adanet_weighted_ensemble = 2.9956787, prediction/mean/adanet/subnetwork = 2.9956787, prediction/mean/adanet/uniform_average_ensemble = 2.9956787\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 15000: /tmp/models/uniform_average_ensemble_baseline/model.ckpt-15000\n",
            "INFO:tensorflow:global_step/sec: 37.3601\n",
            "INFO:tensorflow:loss = 0.023306975, step = 15001 (2.678 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.903\n",
            "INFO:tensorflow:loss = 0.011741974, step = 15101 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.134\n",
            "INFO:tensorflow:loss = 0.039118372, step = 15201 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 636.035\n",
            "INFO:tensorflow:loss = 0.02913019, step = 15301 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 635.91\n",
            "INFO:tensorflow:loss = 0.008382331, step = 15401 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 634.787\n",
            "INFO:tensorflow:loss = 0.018823331, step = 15501 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 631.138\n",
            "INFO:tensorflow:loss = 0.021800134, step = 15601 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.897\n",
            "INFO:tensorflow:loss = 0.023131967, step = 15701 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.681\n",
            "INFO:tensorflow:loss = 0.038506176, step = 15801 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 616.473\n",
            "INFO:tensorflow:loss = 0.0198643, step = 15901 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 642.806\n",
            "INFO:tensorflow:loss = 0.009991525, step = 16001 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 641.667\n",
            "INFO:tensorflow:loss = 0.050554883, step = 16101 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 630.966\n",
            "INFO:tensorflow:loss = 0.016719822, step = 16201 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.985\n",
            "INFO:tensorflow:loss = 0.03370116, step = 16301 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 630.339\n",
            "INFO:tensorflow:loss = 0.012995366, step = 16401 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 640.944\n",
            "INFO:tensorflow:loss = 0.014249171, step = 16501 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.146\n",
            "INFO:tensorflow:loss = 0.012871573, step = 16601 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 636.359\n",
            "INFO:tensorflow:loss = 0.024555627, step = 16701 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 625.833\n",
            "INFO:tensorflow:loss = 0.01776206, step = 16801 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 610.192\n",
            "INFO:tensorflow:loss = 0.02007091, step = 16901 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 615.839\n",
            "INFO:tensorflow:loss = 0.019431155, step = 17001 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.369\n",
            "INFO:tensorflow:loss = 0.014104128, step = 17101 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.25\n",
            "INFO:tensorflow:loss = 0.011531585, step = 17201 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 579.737\n",
            "INFO:tensorflow:loss = 0.020848745, step = 17301 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 565.934\n",
            "INFO:tensorflow:loss = 0.016150339, step = 17401 (0.177 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.662\n",
            "INFO:tensorflow:loss = 0.0133442525, step = 17501 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 615.052\n",
            "INFO:tensorflow:loss = 0.013858032, step = 17601 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 640.295\n",
            "INFO:tensorflow:loss = 0.017426, step = 17701 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 639.832\n",
            "INFO:tensorflow:loss = 0.02293791, step = 17801 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.428\n",
            "INFO:tensorflow:loss = 0.0162404, step = 17901 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 627.805\n",
            "INFO:tensorflow:loss = 0.01923243, step = 18001 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 639.234\n",
            "INFO:tensorflow:loss = 0.018200941, step = 18101 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 633.445\n",
            "INFO:tensorflow:loss = 0.017652484, step = 18201 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 631.098\n",
            "INFO:tensorflow:loss = 0.02050797, step = 18301 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 637.938\n",
            "INFO:tensorflow:loss = 0.024108196, step = 18401 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 627.222\n",
            "INFO:tensorflow:loss = 0.016014796, step = 18501 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 632.964\n",
            "INFO:tensorflow:loss = 0.015239334, step = 18601 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 637.926\n",
            "INFO:tensorflow:loss = 0.013350293, step = 18701 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.464\n",
            "INFO:tensorflow:loss = 0.009045666, step = 18801 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.535\n",
            "INFO:tensorflow:loss = 0.01973171, step = 18901 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 633.573\n",
            "INFO:tensorflow:loss = 0.019966241, step = 19001 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 631.413\n",
            "INFO:tensorflow:loss = 0.026888603, step = 19101 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 629.355\n",
            "INFO:tensorflow:loss = 0.010133859, step = 19201 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 628.235\n",
            "INFO:tensorflow:loss = 0.022864409, step = 19301 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 635.251\n",
            "INFO:tensorflow:loss = 0.015584775, step = 19401 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 643.608\n",
            "INFO:tensorflow:loss = 0.014473073, step = 19501 (0.155 sec)\n",
            "INFO:tensorflow:global_step/sec: 637.898\n",
            "INFO:tensorflow:loss = 0.010564824, step = 19601 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.448\n",
            "INFO:tensorflow:loss = 0.01082034, step = 19701 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 650.326\n",
            "INFO:tensorflow:loss = 0.012072289, step = 19801 (0.154 sec)\n",
            "INFO:tensorflow:global_step/sec: 642.05\n",
            "INFO:tensorflow:loss = 0.0484541, step = 19901 (0.156 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 20000 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-15:57:37\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/model.ckpt-20000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_linear' dict for global step 20000: architecture/adanet/ensembles = \n",
            "W\n",
            "9adanet/iteration_0/ensemble_t0_linear/architecture/adanetB\u0010\u0008\u0007\u0012\u0000B\n",
            "| linear |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.049421377, average_loss/adanet/subnetwork = 0.049421377, average_loss/adanet/uniform_average_ensemble = 0.049421377, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.062442042, loss/adanet/subnetwork = 0.062442042, loss/adanet/uniform_average_ensemble = 0.062442042, prediction/mean/adanet/adanet_weighted_ensemble = 3.105895, prediction/mean/adanet/subnetwork = 3.105895, prediction/mean/adanet/uniform_average_ensemble = 3.105895\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 20000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036273483, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046664447, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.140079, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-15:57:38\n",
            "INFO:tensorflow:Saving dict for global step 20000: average_loss = 0.036273483, average_loss/adanet/adanet_weighted_ensemble = 0.036273483, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, global_step = 20000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.046664447, loss/adanet/adanet_weighted_ensemble = 0.046664447, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean = 3.140079, prediction/mean/adanet/adanet_weighted_ensemble = 3.140079, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 20000: /tmp/models/uniform_average_ensemble_baseline/model.ckpt-20000\n",
            "INFO:tensorflow:Loss for final step: 0.02852798.\n",
            "INFO:tensorflow:Finished training Adanet iteration 0\n",
            "INFO:tensorflow:Beginning bookkeeping phase for iteration 0\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Starting ensemble evaluation for iteration 0\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/model.ckpt-20000\n",
            "WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/adanet/core/estimator.py:729: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "To construct input pipelines, use the `tf.data` module.\n",
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n",
            "INFO:tensorflow:Encountered end of input after 14 evaluations\n",
            "INFO:tensorflow:Computed ensemble metrics: adanet_loss/t0_linear = 0.035089, adanet_loss/t0_1_layer_dnn = 0.016706\n",
            "INFO:tensorflow:Finished ensemble evaluation for iteration 0\n",
            "INFO:tensorflow:'t0_1_layer_dnn' at index 1 is moving onto the next iteration\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/uniform_average_ensemble_baseline/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Warm-starting from: (u'/tmp/models/uniform_average_ensemble_baseline/model.ckpt-20000',)\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/logits/mixture_weight; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: global_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss/local_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/train_op/is_over/is_over_var_fn/is_over_var; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss/biased; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense_1/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense_1/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Overwriting checkpoint with new graph for iteration 1 to /tmp/models/uniform_average_ensemble_baseline/model.ckpt-20000\n",
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n",
            "INFO:tensorflow:Finished bookkeeping phase for iteration 0\n",
            "INFO:tensorflow:Beginning training AdaNet iteration 1\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/uniform_average_ensemble_baseline/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Create CheckpointSaverHook.\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/increment.ckpt-1\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving checkpoints for 20000 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:loss = 0.018871792, step = 20001\n",
            "INFO:tensorflow:global_step/sec: 200.469\n",
            "INFO:tensorflow:loss = 0.025719572, step = 20101 (0.500 sec)\n",
            "INFO:tensorflow:global_step/sec: 589.491\n",
            "INFO:tensorflow:loss = 0.019348014, step = 20201 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 603.106\n",
            "INFO:tensorflow:loss = 0.016309785, step = 20301 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 604.132\n",
            "INFO:tensorflow:loss = 0.021410633, step = 20401 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 593.472\n",
            "INFO:tensorflow:loss = 0.013557139, step = 20501 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 586.83\n",
            "INFO:tensorflow:loss = 0.015202908, step = 20601 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 589.744\n",
            "INFO:tensorflow:loss = 0.010390527, step = 20701 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 583.175\n",
            "INFO:tensorflow:loss = 0.016401442, step = 20801 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 581.362\n",
            "INFO:tensorflow:loss = 0.019048845, step = 20901 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 588.121\n",
            "INFO:tensorflow:loss = 0.022723094, step = 21001 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 587.075\n",
            "INFO:tensorflow:loss = 0.018418074, step = 21101 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 594.279\n",
            "INFO:tensorflow:loss = 0.019738307, step = 21201 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 585.083\n",
            "INFO:tensorflow:loss = 0.01891475, step = 21301 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 591.275\n",
            "INFO:tensorflow:loss = 0.012828046, step = 21401 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 590.918\n",
            "INFO:tensorflow:loss = 0.009291048, step = 21501 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 593.398\n",
            "INFO:tensorflow:loss = 0.008737061, step = 21601 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 589.588\n",
            "INFO:tensorflow:loss = 0.017247716, step = 21701 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 584.361\n",
            "INFO:tensorflow:loss = 0.016145222, step = 21801 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 582.816\n",
            "INFO:tensorflow:loss = 0.01666563, step = 21901 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 589.164\n",
            "INFO:tensorflow:loss = 0.011569293, step = 22001 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 597.218\n",
            "INFO:tensorflow:loss = 0.026046487, step = 22101 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 587.461\n",
            "INFO:tensorflow:loss = 0.017956626, step = 22201 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 596.249\n",
            "INFO:tensorflow:loss = 0.01227755, step = 22301 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 599.984\n",
            "INFO:tensorflow:loss = 0.017292287, step = 22401 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 600.266\n",
            "INFO:tensorflow:loss = 0.022227846, step = 22501 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 591.894\n",
            "INFO:tensorflow:loss = 0.012348479, step = 22601 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 593.324\n",
            "INFO:tensorflow:loss = 0.02043081, step = 22701 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 513.545\n",
            "INFO:tensorflow:loss = 0.013893949, step = 22801 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 593.102\n",
            "INFO:tensorflow:loss = 0.012673097, step = 22901 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 573.415\n",
            "INFO:tensorflow:loss = 0.014353249, step = 23001 (0.174 sec)\n",
            "INFO:tensorflow:global_step/sec: 584.795\n",
            "INFO:tensorflow:loss = 0.0069335564, step = 23101 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 591.223\n",
            "INFO:tensorflow:loss = 0.018832773, step = 23201 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 585.727\n",
            "INFO:tensorflow:loss = 0.02307523, step = 23301 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 603.883\n",
            "INFO:tensorflow:loss = 0.017754633, step = 23401 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 587.331\n",
            "INFO:tensorflow:loss = 0.023659421, step = 23501 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 582.543\n",
            "INFO:tensorflow:loss = 0.011922585, step = 23601 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 594.696\n",
            "INFO:tensorflow:loss = 0.018203132, step = 23701 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 594.619\n",
            "INFO:tensorflow:loss = 0.010016835, step = 23801 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 596.122\n",
            "INFO:tensorflow:loss = 0.01064863, step = 23901 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 587.596\n",
            "INFO:tensorflow:loss = 0.014820645, step = 24001 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 535.152\n",
            "INFO:tensorflow:loss = 0.01900113, step = 24101 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 582.068\n",
            "INFO:tensorflow:loss = 0.012903263, step = 24201 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 580.376\n",
            "INFO:tensorflow:loss = 0.009616712, step = 24301 (0.176 sec)\n",
            "INFO:tensorflow:global_step/sec: 580.229\n",
            "INFO:tensorflow:loss = 0.015747245, step = 24401 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 579.478\n",
            "INFO:tensorflow:loss = 0.012267083, step = 24501 (0.173 sec)\n",
            "INFO:tensorflow:global_step/sec: 585.507\n",
            "INFO:tensorflow:loss = 0.015497938, step = 24601 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 586.834\n",
            "INFO:tensorflow:loss = 0.0133473, step = 24701 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 592.782\n",
            "INFO:tensorflow:loss = 0.011023393, step = 24801 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 587.017\n",
            "INFO:tensorflow:loss = 0.014582325, step = 24901 (0.170 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 25000 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/uniform_average_ensemble_baseline/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-15:57:59\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/model.ckpt-25000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 25000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036273483, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046664447, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.140079, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving candidate 't1_1_layer_dnn' dict for global step 25000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_1_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.04408999, average_loss/adanet/subnetwork = 0.05592768, average_loss/adanet/uniform_average_ensemble = 0.04408999, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.06125318, loss/adanet/subnetwork = 0.081614695, loss/adanet/uniform_average_ensemble = 0.06125318, prediction/mean/adanet/adanet_weighted_ensemble = 3.15282, prediction/mean/adanet/subnetwork = 3.1655612, prediction/mean/adanet/uniform_average_ensemble = 3.15282\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 25000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03759182, average_loss/adanet/subnetwork = 0.042800613, average_loss/adanet/uniform_average_ensemble = 0.03759182, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.04678309, loss/adanet/subnetwork = 0.050511263, loss/adanet/uniform_average_ensemble = 0.04678309, prediction/mean/adanet/adanet_weighted_ensemble = 3.1083033, prediction/mean/adanet/subnetwork = 3.0765278, prediction/mean/adanet/uniform_average_ensemble = 3.1083033\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-15:58:01\n",
            "INFO:tensorflow:Saving dict for global step 25000: average_loss = 0.03759182, average_loss/adanet/adanet_weighted_ensemble = 0.03759182, average_loss/adanet/subnetwork = 0.042800613, average_loss/adanet/uniform_average_ensemble = 0.03759182, global_step = 25000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.04678309, loss/adanet/adanet_weighted_ensemble = 0.04678309, loss/adanet/subnetwork = 0.050511263, loss/adanet/uniform_average_ensemble = 0.04678309, prediction/mean = 3.1083033, prediction/mean/adanet/adanet_weighted_ensemble = 3.1083033, prediction/mean/adanet/subnetwork = 3.0765278, prediction/mean/adanet/uniform_average_ensemble = 3.1083033\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 25000: /tmp/models/uniform_average_ensemble_baseline/model.ckpt-25000\n",
            "INFO:tensorflow:global_step/sec: 19.0671\n",
            "INFO:tensorflow:loss = 0.008009167, step = 25001 (5.245 sec)\n",
            "INFO:tensorflow:global_step/sec: 593.979\n",
            "INFO:tensorflow:loss = 0.016955424, step = 25101 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 577.808\n",
            "INFO:tensorflow:loss = 0.01730625, step = 25201 (0.174 sec)\n",
            "INFO:tensorflow:global_step/sec: 597.314\n",
            "INFO:tensorflow:loss = 0.018815871, step = 25301 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 587.661\n",
            "INFO:tensorflow:loss = 0.009227458, step = 25401 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 592.695\n",
            "INFO:tensorflow:loss = 0.006362489, step = 25501 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 596.182\n",
            "INFO:tensorflow:loss = 0.0146550685, step = 25601 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 592.158\n",
            "INFO:tensorflow:loss = 0.012894993, step = 25701 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 407.169\n",
            "INFO:tensorflow:loss = 0.015100146, step = 25801 (0.246 sec)\n",
            "INFO:tensorflow:global_step/sec: 510.72\n",
            "INFO:tensorflow:loss = 0.011555306, step = 25901 (0.203 sec)\n",
            "INFO:tensorflow:global_step/sec: 472.84\n",
            "INFO:tensorflow:loss = 0.008945199, step = 26001 (0.205 sec)\n",
            "INFO:tensorflow:global_step/sec: 604.91\n",
            "INFO:tensorflow:loss = 0.016919287, step = 26101 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 576.335\n",
            "INFO:tensorflow:loss = 0.013710671, step = 26201 (0.173 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.158\n",
            "INFO:tensorflow:loss = 0.007565688, step = 26301 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 598.004\n",
            "INFO:tensorflow:loss = 0.013592415, step = 26401 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 595.377\n",
            "INFO:tensorflow:loss = 0.014497319, step = 26501 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 594.738\n",
            "INFO:tensorflow:loss = 0.02051242, step = 26601 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 592.61\n",
            "INFO:tensorflow:loss = 0.014250898, step = 26701 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 601.587\n",
            "INFO:tensorflow:loss = 0.009928584, step = 26801 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 594.265\n",
            "INFO:tensorflow:loss = 0.005977773, step = 26901 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 605.232\n",
            "INFO:tensorflow:loss = 0.009271427, step = 27001 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 603.224\n",
            "INFO:tensorflow:loss = 0.011318848, step = 27101 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 590.368\n",
            "INFO:tensorflow:loss = 0.009759569, step = 27201 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 599.743\n",
            "INFO:tensorflow:loss = 0.016437937, step = 27301 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 590.325\n",
            "INFO:tensorflow:loss = 0.009727536, step = 27401 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 591.713\n",
            "INFO:tensorflow:loss = 0.006248885, step = 27501 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 594.926\n",
            "INFO:tensorflow:loss = 0.009168317, step = 27601 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 598.312\n",
            "INFO:tensorflow:loss = 0.02344711, step = 27701 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 590.089\n",
            "INFO:tensorflow:loss = 0.014392592, step = 27801 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.074\n",
            "INFO:tensorflow:loss = 0.0071986727, step = 27901 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 588.114\n",
            "INFO:tensorflow:loss = 0.018193342, step = 28001 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 596.712\n",
            "INFO:tensorflow:loss = 0.01741543, step = 28101 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 589.737\n",
            "INFO:tensorflow:loss = 0.017680477, step = 28201 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 598.262\n",
            "INFO:tensorflow:loss = 0.015683174, step = 28301 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 582.931\n",
            "INFO:tensorflow:loss = 0.008351777, step = 28401 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 602.427\n",
            "INFO:tensorflow:loss = 0.007883148, step = 28501 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 596.46\n",
            "INFO:tensorflow:loss = 0.011117501, step = 28601 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 472.224\n",
            "INFO:tensorflow:loss = 0.011717734, step = 28701 (0.212 sec)\n",
            "INFO:tensorflow:global_step/sec: 589.348\n",
            "INFO:tensorflow:loss = 0.020395812, step = 28801 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 589.599\n",
            "INFO:tensorflow:loss = 0.019548718, step = 28901 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 598.982\n",
            "INFO:tensorflow:loss = 0.019873034, step = 29001 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 595.55\n",
            "INFO:tensorflow:loss = 0.016073726, step = 29101 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 586.692\n",
            "INFO:tensorflow:loss = 0.012113925, step = 29201 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 603.733\n",
            "INFO:tensorflow:loss = 0.0066889776, step = 29301 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 592.411\n",
            "INFO:tensorflow:loss = 0.014553456, step = 29401 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 592.143\n",
            "INFO:tensorflow:loss = 0.017371256, step = 29501 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 595.515\n",
            "INFO:tensorflow:loss = 0.00996026, step = 29601 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 592.958\n",
            "INFO:tensorflow:loss = 0.008428328, step = 29701 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 601.243\n",
            "INFO:tensorflow:loss = 0.012243295, step = 29801 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 594.859\n",
            "INFO:tensorflow:loss = 0.015630722, step = 29901 (0.168 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 30000 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/uniform_average_ensemble_baseline/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-15:58:12\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/model.ckpt-30000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 30000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036273483, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046664447, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.140079, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving candidate 't1_1_layer_dnn' dict for global step 30000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_1_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.039409403, average_loss/adanet/subnetwork = 0.047193587, average_loss/adanet/uniform_average_ensemble = 0.039409403, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.05252158, loss/adanet/subnetwork = 0.0635132, loss/adanet/uniform_average_ensemble = 0.05252158, prediction/mean/adanet/adanet_weighted_ensemble = 3.1105392, prediction/mean/adanet/subnetwork = 3.0809996, prediction/mean/adanet/uniform_average_ensemble = 3.1105392\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 30000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035760086, average_loss/adanet/subnetwork = 0.04406667, average_loss/adanet/uniform_average_ensemble = 0.035760086, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.044707134, loss/adanet/subnetwork = 0.052304745, loss/adanet/uniform_average_ensemble = 0.044707134, prediction/mean/adanet/adanet_weighted_ensemble = 3.0812702, prediction/mean/adanet/subnetwork = 3.0224612, prediction/mean/adanet/uniform_average_ensemble = 3.0812702\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-15:58:14\n",
            "INFO:tensorflow:Saving dict for global step 30000: average_loss = 0.035760086, average_loss/adanet/adanet_weighted_ensemble = 0.035760086, average_loss/adanet/subnetwork = 0.04406667, average_loss/adanet/uniform_average_ensemble = 0.035760086, global_step = 30000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.044707134, loss/adanet/adanet_weighted_ensemble = 0.044707134, loss/adanet/subnetwork = 0.052304745, loss/adanet/uniform_average_ensemble = 0.044707134, prediction/mean = 3.0812702, prediction/mean/adanet/adanet_weighted_ensemble = 3.0812702, prediction/mean/adanet/subnetwork = 3.0224612, prediction/mean/adanet/uniform_average_ensemble = 3.0812702\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 30000: /tmp/models/uniform_average_ensemble_baseline/model.ckpt-30000\n",
            "INFO:tensorflow:global_step/sec: 22.5421\n",
            "INFO:tensorflow:loss = 0.013591687, step = 30001 (4.441 sec)\n",
            "INFO:tensorflow:global_step/sec: 566.197\n",
            "INFO:tensorflow:loss = 0.0045358343, step = 30101 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 579.26\n",
            "INFO:tensorflow:loss = 0.014999437, step = 30201 (0.173 sec)\n",
            "INFO:tensorflow:global_step/sec: 572.132\n",
            "INFO:tensorflow:loss = 0.01080295, step = 30301 (0.175 sec)\n",
            "INFO:tensorflow:global_step/sec: 587.003\n",
            "INFO:tensorflow:loss = 0.0065587605, step = 30401 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 590.915\n",
            "INFO:tensorflow:loss = 0.0048322054, step = 30501 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 570.103\n",
            "INFO:tensorflow:loss = 0.008199327, step = 30601 (0.176 sec)\n",
            "INFO:tensorflow:global_step/sec: 551.155\n",
            "INFO:tensorflow:loss = 0.0085193785, step = 30701 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 580.481\n",
            "INFO:tensorflow:loss = 0.01113713, step = 30801 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 581.733\n",
            "INFO:tensorflow:loss = 0.014991447, step = 30901 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 593.302\n",
            "INFO:tensorflow:loss = 0.0070897415, step = 31001 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 590.818\n",
            "INFO:tensorflow:loss = 0.01131255, step = 31101 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 582.55\n",
            "INFO:tensorflow:loss = 0.008938184, step = 31201 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 597.368\n",
            "INFO:tensorflow:loss = 0.011916299, step = 31301 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 587.348\n",
            "INFO:tensorflow:loss = 0.0052154153, step = 31401 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 585.868\n",
            "INFO:tensorflow:loss = 0.021725576, step = 31501 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 596.797\n",
            "INFO:tensorflow:loss = 0.011893438, step = 31601 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 585.864\n",
            "INFO:tensorflow:loss = 0.013616124, step = 31701 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 567.621\n",
            "INFO:tensorflow:loss = 0.006440956, step = 31801 (0.176 sec)\n",
            "INFO:tensorflow:global_step/sec: 583.305\n",
            "INFO:tensorflow:loss = 0.012435072, step = 31901 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 587.337\n",
            "INFO:tensorflow:loss = 0.017083135, step = 32001 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 473.507\n",
            "INFO:tensorflow:loss = 0.01219175, step = 32101 (0.211 sec)\n",
            "INFO:tensorflow:global_step/sec: 595.717\n",
            "INFO:tensorflow:loss = 0.009562939, step = 32201 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 595.372\n",
            "INFO:tensorflow:loss = 0.008798393, step = 32301 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 576.203\n",
            "INFO:tensorflow:loss = 0.010170335, step = 32401 (0.174 sec)\n",
            "INFO:tensorflow:global_step/sec: 586.146\n",
            "INFO:tensorflow:loss = 0.01471898, step = 32501 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 596.533\n",
            "INFO:tensorflow:loss = 0.007682061, step = 32601 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 589.397\n",
            "INFO:tensorflow:loss = 0.016532334, step = 32701 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 583.989\n",
            "INFO:tensorflow:loss = 0.015530237, step = 32801 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 600.947\n",
            "INFO:tensorflow:loss = 0.010471679, step = 32901 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 598.444\n",
            "INFO:tensorflow:loss = 0.014978467, step = 33001 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 596.591\n",
            "INFO:tensorflow:loss = 0.011348784, step = 33101 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 588.087\n",
            "INFO:tensorflow:loss = 0.008112838, step = 33201 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 588.879\n",
            "INFO:tensorflow:loss = 0.006526803, step = 33301 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 589.247\n",
            "INFO:tensorflow:loss = 0.009947769, step = 33401 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 579.593\n",
            "INFO:tensorflow:loss = 0.011564908, step = 33501 (0.173 sec)\n",
            "INFO:tensorflow:global_step/sec: 581.957\n",
            "INFO:tensorflow:loss = 0.012979926, step = 33601 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 585.329\n",
            "INFO:tensorflow:loss = 0.005131601, step = 33701 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 590.636\n",
            "INFO:tensorflow:loss = 0.012794479, step = 33801 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 595.213\n",
            "INFO:tensorflow:loss = 0.012131458, step = 33901 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 591.559\n",
            "INFO:tensorflow:loss = 0.006763425, step = 34001 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 586.899\n",
            "INFO:tensorflow:loss = 0.011409021, step = 34101 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 579.086\n",
            "INFO:tensorflow:loss = 0.010845594, step = 34201 (0.173 sec)\n",
            "INFO:tensorflow:global_step/sec: 581.963\n",
            "INFO:tensorflow:loss = 0.009460603, step = 34301 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 589.084\n",
            "INFO:tensorflow:loss = 0.007882768, step = 34401 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 603.479\n",
            "INFO:tensorflow:loss = 0.00823106, step = 34501 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 593.595\n",
            "INFO:tensorflow:loss = 0.00962188, step = 34601 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 589.377\n",
            "INFO:tensorflow:loss = 0.012230248, step = 34701 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 597.425\n",
            "INFO:tensorflow:loss = 0.015466228, step = 34801 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 596.937\n",
            "INFO:tensorflow:loss = 0.008023506, step = 34901 (0.167 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 35000 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/uniform_average_ensemble_baseline/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-15:58:25\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/model.ckpt-35000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 35000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036273483, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046664447, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.140079, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving candidate 't1_1_layer_dnn' dict for global step 35000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_1_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.037198033, average_loss/adanet/subnetwork = 0.047201026, average_loss/adanet/uniform_average_ensemble = 0.037198033, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.045761347, loss/adanet/subnetwork = 0.054078415, loss/adanet/uniform_average_ensemble = 0.045761347, prediction/mean/adanet/adanet_weighted_ensemble = 3.0827281, prediction/mean/adanet/subnetwork = 3.0253773, prediction/mean/adanet/uniform_average_ensemble = 3.0827281\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 35000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.0361695, average_loss/adanet/subnetwork = 0.04869274, average_loss/adanet/uniform_average_ensemble = 0.0361695, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.04201368, loss/adanet/subnetwork = 0.053367704, loss/adanet/uniform_average_ensemble = 0.04201368, prediction/mean/adanet/adanet_weighted_ensemble = 3.0712888, prediction/mean/adanet/subnetwork = 3.0024989, prediction/mean/adanet/uniform_average_ensemble = 3.0712888\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-15:58:26\n",
            "INFO:tensorflow:Saving dict for global step 35000: average_loss = 0.0361695, average_loss/adanet/adanet_weighted_ensemble = 0.0361695, average_loss/adanet/subnetwork = 0.04869274, average_loss/adanet/uniform_average_ensemble = 0.0361695, global_step = 35000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.04201368, loss/adanet/adanet_weighted_ensemble = 0.04201368, loss/adanet/subnetwork = 0.053367704, loss/adanet/uniform_average_ensemble = 0.04201368, prediction/mean = 3.0712888, prediction/mean/adanet/adanet_weighted_ensemble = 3.0712888, prediction/mean/adanet/subnetwork = 3.0024989, prediction/mean/adanet/uniform_average_ensemble = 3.0712888\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 35000: /tmp/models/uniform_average_ensemble_baseline/model.ckpt-35000\n",
            "INFO:tensorflow:global_step/sec: 22.652\n",
            "INFO:tensorflow:loss = 0.015470884, step = 35001 (4.415 sec)\n",
            "INFO:tensorflow:global_step/sec: 592.53\n",
            "INFO:tensorflow:loss = 0.008049628, step = 35101 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.981\n",
            "INFO:tensorflow:loss = 0.010416111, step = 35201 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 597.533\n",
            "INFO:tensorflow:loss = 0.013864119, step = 35301 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.523\n",
            "INFO:tensorflow:loss = 0.0070619783, step = 35401 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 469.953\n",
            "INFO:tensorflow:loss = 0.014238533, step = 35501 (0.213 sec)\n",
            "INFO:tensorflow:global_step/sec: 573.536\n",
            "INFO:tensorflow:loss = 0.014644979, step = 35601 (0.174 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.681\n",
            "INFO:tensorflow:loss = 0.007587858, step = 35701 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.74\n",
            "INFO:tensorflow:loss = 0.015924843, step = 35801 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 577.952\n",
            "INFO:tensorflow:loss = 0.007015991, step = 35901 (0.173 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.156\n",
            "INFO:tensorflow:loss = 0.007330937, step = 36001 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 568.414\n",
            "INFO:tensorflow:loss = 0.01225366, step = 36101 (0.176 sec)\n",
            "INFO:tensorflow:global_step/sec: 562.034\n",
            "INFO:tensorflow:loss = 0.008415307, step = 36201 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 578.072\n",
            "INFO:tensorflow:loss = 0.0060964134, step = 36301 (0.173 sec)\n",
            "INFO:tensorflow:global_step/sec: 572.816\n",
            "INFO:tensorflow:loss = 0.0065386053, step = 36401 (0.175 sec)\n",
            "INFO:tensorflow:global_step/sec: 574.875\n",
            "INFO:tensorflow:loss = 0.007027081, step = 36501 (0.174 sec)\n",
            "INFO:tensorflow:global_step/sec: 582.941\n",
            "INFO:tensorflow:loss = 0.008016022, step = 36601 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 585.946\n",
            "INFO:tensorflow:loss = 0.013963519, step = 36701 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 580.31\n",
            "INFO:tensorflow:loss = 0.008944197, step = 36801 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 588.328\n",
            "INFO:tensorflow:loss = 0.009293896, step = 36901 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 587.89\n",
            "INFO:tensorflow:loss = 0.011544491, step = 37001 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 585.586\n",
            "INFO:tensorflow:loss = 0.0144116, step = 37101 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 574.498\n",
            "INFO:tensorflow:loss = 0.0058088717, step = 37201 (0.174 sec)\n",
            "INFO:tensorflow:global_step/sec: 599.42\n",
            "INFO:tensorflow:loss = 0.005003029, step = 37301 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 594.406\n",
            "INFO:tensorflow:loss = 0.0049871863, step = 37401 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 597.597\n",
            "INFO:tensorflow:loss = 0.009269521, step = 37501 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 572.629\n",
            "INFO:tensorflow:loss = 0.010450633, step = 37601 (0.175 sec)\n",
            "INFO:tensorflow:global_step/sec: 583.315\n",
            "INFO:tensorflow:loss = 0.007788347, step = 37701 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 598.24\n",
            "INFO:tensorflow:loss = 0.015467301, step = 37801 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 586.468\n",
            "INFO:tensorflow:loss = 0.0066822353, step = 37901 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 574.218\n",
            "INFO:tensorflow:loss = 0.011355821, step = 38001 (0.174 sec)\n",
            "INFO:tensorflow:global_step/sec: 597.797\n",
            "INFO:tensorflow:loss = 0.012325816, step = 38101 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 591.786\n",
            "INFO:tensorflow:loss = 0.013065682, step = 38201 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 581.432\n",
            "INFO:tensorflow:loss = 0.014039053, step = 38301 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 527.312\n",
            "INFO:tensorflow:loss = 0.011183753, step = 38401 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 606.278\n",
            "INFO:tensorflow:loss = 0.011565153, step = 38501 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 604.292\n",
            "INFO:tensorflow:loss = 0.009946515, step = 38601 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 596.847\n",
            "INFO:tensorflow:loss = 0.0054588886, step = 38701 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 573.838\n",
            "INFO:tensorflow:loss = 0.0065397024, step = 38801 (0.174 sec)\n",
            "INFO:tensorflow:global_step/sec: 581.011\n",
            "INFO:tensorflow:loss = 0.013163909, step = 38901 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 581.324\n",
            "INFO:tensorflow:loss = 0.014859106, step = 39001 (0.172 sec)\n",
            "INFO:tensorflow:global_step/sec: 579.522\n",
            "INFO:tensorflow:loss = 0.009422213, step = 39101 (0.173 sec)\n",
            "INFO:tensorflow:global_step/sec: 590.316\n",
            "INFO:tensorflow:loss = 0.0065256944, step = 39201 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 588.582\n",
            "INFO:tensorflow:loss = 0.0076505356, step = 39301 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 585.086\n",
            "INFO:tensorflow:loss = 0.00535091, step = 39401 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 578.553\n",
            "INFO:tensorflow:loss = 0.006468159, step = 39501 (0.173 sec)\n",
            "INFO:tensorflow:global_step/sec: 589.007\n",
            "INFO:tensorflow:loss = 0.00793083, step = 39601 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 588.288\n",
            "INFO:tensorflow:loss = 0.0058432827, step = 39701 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 586.238\n",
            "INFO:tensorflow:loss = 0.01980201, step = 39801 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 588.315\n",
            "INFO:tensorflow:loss = 0.018302646, step = 39901 (0.170 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 40000 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/uniform_average_ensemble_baseline/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-15:58:38\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/model.ckpt-40000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 40000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036273483, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046664447, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.140079, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving candidate 't1_1_layer_dnn' dict for global step 40000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_1_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036194302, average_loss/adanet/subnetwork = 0.03862485, average_loss/adanet/uniform_average_ensemble = 0.036194302, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.048834026, loss/adanet/subnetwork = 0.05337928, loss/adanet/uniform_average_ensemble = 0.048834026, prediction/mean/adanet/adanet_weighted_ensemble = 3.153415, prediction/mean/adanet/subnetwork = 3.1667511, prediction/mean/adanet/uniform_average_ensemble = 3.153415\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 40000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036551207, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.049044557, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.143447, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-15:58:39\n",
            "INFO:tensorflow:Saving dict for global step 40000: average_loss = 0.036551207, average_loss/adanet/adanet_weighted_ensemble = 0.036551207, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, global_step = 40000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.049044557, loss/adanet/adanet_weighted_ensemble = 0.049044557, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean = 3.143447, prediction/mean/adanet/adanet_weighted_ensemble = 3.143447, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 40000: /tmp/models/uniform_average_ensemble_baseline/model.ckpt-40000\n",
            "INFO:tensorflow:Loss for final step: 0.007572856.\n",
            "INFO:tensorflow:Finished training Adanet iteration 1\n",
            "INFO:tensorflow:Beginning bookkeeping phase for iteration 1\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/uniform_average_ensemble_baseline/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Starting ensemble evaluation for iteration 1\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/model.ckpt-40000\n",
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n",
            "INFO:tensorflow:Encountered end of input after 14 evaluations\n",
            "INFO:tensorflow:Computed ensemble metrics: adanet_loss/t0_1_layer_dnn = 0.016706, adanet_loss/t1_1_layer_dnn = 0.016541, adanet_loss/t1_2_layer_dnn = 0.013700\n",
            "INFO:tensorflow:Finished ensemble evaluation for iteration 1\n",
            "INFO:tensorflow:'t1_2_layer_dnn' at index 2 is moving onto the next iteration\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/uniform_average_ensemble_baseline/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Warm-starting from: (u'/tmp/models/uniform_average_ensemble_baseline/model.ckpt-40000',)\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss/biased; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense_1/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense_1/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t1_2_layer_dnn/adanet_loss; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: global_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss/local_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense_1/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_0/logits/mixture_weight; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t0_1_layer_dnn/adanet_loss; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense_2/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t1_2_layer_dnn/adanet/iteration_1/candidate_t1_2_layer_dnn/adanet_loss/biased; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/logits/mixture_weight; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense_1/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/logits/mixture_weight; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t1_2_layer_dnn/adanet/iteration_1/candidate_t1_2_layer_dnn/adanet_loss/local_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t0_1_layer_dnn/adanet/iteration_1/candidate_t0_1_layer_dnn/adanet_loss/biased; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t0_1_layer_dnn/adanet/iteration_1/candidate_t0_1_layer_dnn/adanet_loss/local_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense_2/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/train_op/is_over/is_over_var_fn/is_over_var; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/train_op/is_over/is_over_var_fn/is_over_var; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Overwriting checkpoint with new graph for iteration 2 to /tmp/models/uniform_average_ensemble_baseline/model.ckpt-40000\n",
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n",
            "INFO:tensorflow:Finished bookkeeping phase for iteration 1\n",
            "INFO:tensorflow:Beginning training AdaNet iteration 2\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/uniform_average_ensemble_baseline/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Create CheckpointSaverHook.\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/increment.ckpt-2\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving checkpoints for 40000 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:loss = 0.016760603, step = 40001\n",
            "INFO:tensorflow:global_step/sec: 161.711\n",
            "INFO:tensorflow:loss = 0.017622853, step = 40101 (0.620 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.764\n",
            "INFO:tensorflow:loss = 0.017423391, step = 40201 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 530.31\n",
            "INFO:tensorflow:loss = 0.013957158, step = 40301 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.429\n",
            "INFO:tensorflow:loss = 0.016208405, step = 40401 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 540.28\n",
            "INFO:tensorflow:loss = 0.011129519, step = 40501 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 532.697\n",
            "INFO:tensorflow:loss = 0.010117756, step = 40601 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 531.279\n",
            "INFO:tensorflow:loss = 0.010820568, step = 40701 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 519.378\n",
            "INFO:tensorflow:loss = 0.014035864, step = 40801 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.947\n",
            "INFO:tensorflow:loss = 0.014903186, step = 40901 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 535.958\n",
            "INFO:tensorflow:loss = 0.02035465, step = 41001 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 538.74\n",
            "INFO:tensorflow:loss = 0.015567474, step = 41101 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 528.572\n",
            "INFO:tensorflow:loss = 0.017009486, step = 41201 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.465\n",
            "INFO:tensorflow:loss = 0.009715016, step = 41301 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 531.132\n",
            "INFO:tensorflow:loss = 0.009901937, step = 41401 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.786\n",
            "INFO:tensorflow:loss = 0.0070146145, step = 41501 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 534.262\n",
            "INFO:tensorflow:loss = 0.0071561667, step = 41601 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 533.186\n",
            "INFO:tensorflow:loss = 0.01575845, step = 41701 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 542.967\n",
            "INFO:tensorflow:loss = 0.010595178, step = 41801 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 544.52\n",
            "INFO:tensorflow:loss = 0.011111792, step = 41901 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 543.659\n",
            "INFO:tensorflow:loss = 0.009603186, step = 42001 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 453.385\n",
            "INFO:tensorflow:loss = 0.021185696, step = 42101 (0.220 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.666\n",
            "INFO:tensorflow:loss = 0.008712892, step = 42201 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.482\n",
            "INFO:tensorflow:loss = 0.009855529, step = 42301 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 529.235\n",
            "INFO:tensorflow:loss = 0.012907856, step = 42401 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 535.14\n",
            "INFO:tensorflow:loss = 0.019562948, step = 42501 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.013\n",
            "INFO:tensorflow:loss = 0.006664275, step = 42601 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 554.275\n",
            "INFO:tensorflow:loss = 0.01684586, step = 42701 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 532.632\n",
            "INFO:tensorflow:loss = 0.008509669, step = 42801 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.134\n",
            "INFO:tensorflow:loss = 0.0077931248, step = 42901 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.22\n",
            "INFO:tensorflow:loss = 0.012203712, step = 43001 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 540.464\n",
            "INFO:tensorflow:loss = 0.011617619, step = 43101 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 543.366\n",
            "INFO:tensorflow:loss = 0.0117117185, step = 43201 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 521.831\n",
            "INFO:tensorflow:loss = 0.016412482, step = 43301 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.357\n",
            "INFO:tensorflow:loss = 0.01130664, step = 43401 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 543.652\n",
            "INFO:tensorflow:loss = 0.02288478, step = 43501 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 534.574\n",
            "INFO:tensorflow:loss = 0.008110959, step = 43601 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.828\n",
            "INFO:tensorflow:loss = 0.009784792, step = 43701 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 544.422\n",
            "INFO:tensorflow:loss = 0.008560029, step = 43801 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.026\n",
            "INFO:tensorflow:loss = 0.006023009, step = 43901 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.002\n",
            "INFO:tensorflow:loss = 0.010378089, step = 44001 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.783\n",
            "INFO:tensorflow:loss = 0.015744004, step = 44101 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 532.793\n",
            "INFO:tensorflow:loss = 0.010757565, step = 44201 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.834\n",
            "INFO:tensorflow:loss = 0.006941431, step = 44301 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.377\n",
            "INFO:tensorflow:loss = 0.0107949525, step = 44401 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 532.586\n",
            "INFO:tensorflow:loss = 0.010741797, step = 44501 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 535.989\n",
            "INFO:tensorflow:loss = 0.01251184, step = 44601 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 543.753\n",
            "INFO:tensorflow:loss = 0.008878224, step = 44701 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 458.468\n",
            "INFO:tensorflow:loss = 0.005082663, step = 44801 (0.218 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.484\n",
            "INFO:tensorflow:loss = 0.010290976, step = 44901 (0.186 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 45000 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/uniform_average_ensemble_baseline/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-15:59:07\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/model.ckpt-45000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 45000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036551207, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.049044557, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.143447, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving candidate 't2_2_layer_dnn' dict for global step 45000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_2_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.038430028, average_loss/adanet/subnetwork = 0.047327004, average_loss/adanet/uniform_average_ensemble = 0.038430035, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.053052317, loss/adanet/subnetwork = 0.065781765, loss/adanet/uniform_average_ensemble = 0.05305232, prediction/mean/adanet/adanet_weighted_ensemble = 3.124987, prediction/mean/adanet/subnetwork = 3.0880663, prediction/mean/adanet/uniform_average_ensemble = 3.124987\n",
            "INFO:tensorflow:Saving candidate 't2_3_layer_dnn' dict for global step 45000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_3_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 3_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.037772827, average_loss/adanet/subnetwork = 0.044495832, average_loss/adanet/uniform_average_ensemble = 0.037772827, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.051192492, loss/adanet/subnetwork = 0.059036292, loss/adanet/uniform_average_ensemble = 0.051192496, prediction/mean/adanet/adanet_weighted_ensemble = 3.1283543, prediction/mean/adanet/subnetwork = 3.098168, prediction/mean/adanet/uniform_average_ensemble = 3.1283538\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-15:59:10\n",
            "INFO:tensorflow:Saving dict for global step 45000: average_loss = 0.037772827, average_loss/adanet/adanet_weighted_ensemble = 0.037772827, average_loss/adanet/subnetwork = 0.044495832, average_loss/adanet/uniform_average_ensemble = 0.037772827, global_step = 45000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.051192492, loss/adanet/adanet_weighted_ensemble = 0.051192492, loss/adanet/subnetwork = 0.059036292, loss/adanet/uniform_average_ensemble = 0.051192496, prediction/mean = 3.1283543, prediction/mean/adanet/adanet_weighted_ensemble = 3.1283543, prediction/mean/adanet/subnetwork = 3.098168, prediction/mean/adanet/uniform_average_ensemble = 3.1283538\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 45000: /tmp/models/uniform_average_ensemble_baseline/model.ckpt-45000\n",
            "INFO:tensorflow:global_step/sec: 13.7787\n",
            "INFO:tensorflow:loss = 0.0072410614, step = 45001 (7.260 sec)\n",
            "INFO:tensorflow:global_step/sec: 521.804\n",
            "INFO:tensorflow:loss = 0.007733005, step = 45101 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.065\n",
            "INFO:tensorflow:loss = 0.008012895, step = 45201 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 538.587\n",
            "INFO:tensorflow:loss = 0.0155502595, step = 45301 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.615\n",
            "INFO:tensorflow:loss = 0.0061861845, step = 45401 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.029\n",
            "INFO:tensorflow:loss = 0.005914101, step = 45501 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.537\n",
            "INFO:tensorflow:loss = 0.012771077, step = 45601 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 534.32\n",
            "INFO:tensorflow:loss = 0.008873316, step = 45701 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 542.758\n",
            "INFO:tensorflow:loss = 0.012681794, step = 45801 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 532.161\n",
            "INFO:tensorflow:loss = 0.008853018, step = 45901 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.114\n",
            "INFO:tensorflow:loss = 0.007390925, step = 46001 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 508.122\n",
            "INFO:tensorflow:loss = 0.013824654, step = 46101 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 513.904\n",
            "INFO:tensorflow:loss = 0.013778057, step = 46201 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.468\n",
            "INFO:tensorflow:loss = 0.0057471856, step = 46301 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 367.536\n",
            "INFO:tensorflow:loss = 0.010059498, step = 46401 (0.272 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.3\n",
            "INFO:tensorflow:loss = 0.017992288, step = 46501 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 528.125\n",
            "INFO:tensorflow:loss = 0.009824095, step = 46601 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.505\n",
            "INFO:tensorflow:loss = 0.012173165, step = 46701 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 525.572\n",
            "INFO:tensorflow:loss = 0.010283058, step = 46801 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 533.282\n",
            "INFO:tensorflow:loss = 0.0055250293, step = 46901 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 535.618\n",
            "INFO:tensorflow:loss = 0.009960668, step = 47001 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.859\n",
            "INFO:tensorflow:loss = 0.009459639, step = 47101 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 535.34\n",
            "INFO:tensorflow:loss = 0.011271746, step = 47201 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 528.207\n",
            "INFO:tensorflow:loss = 0.014637608, step = 47301 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 524.75\n",
            "INFO:tensorflow:loss = 0.01058808, step = 47401 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 538.468\n",
            "INFO:tensorflow:loss = 0.008776734, step = 47501 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 530.625\n",
            "INFO:tensorflow:loss = 0.0070862733, step = 47601 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.773\n",
            "INFO:tensorflow:loss = 0.013936609, step = 47701 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 533.965\n",
            "INFO:tensorflow:loss = 0.017612657, step = 47801 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 534.436\n",
            "INFO:tensorflow:loss = 0.006777015, step = 47901 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 531.234\n",
            "INFO:tensorflow:loss = 0.017204827, step = 48001 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 534.068\n",
            "INFO:tensorflow:loss = 0.01665388, step = 48101 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 542.012\n",
            "INFO:tensorflow:loss = 0.014966566, step = 48201 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.178\n",
            "INFO:tensorflow:loss = 0.012076043, step = 48301 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 529.987\n",
            "INFO:tensorflow:loss = 0.0058982326, step = 48401 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 533.479\n",
            "INFO:tensorflow:loss = 0.005956145, step = 48501 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 535.865\n",
            "INFO:tensorflow:loss = 0.0092611965, step = 48601 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.325\n",
            "INFO:tensorflow:loss = 0.011824546, step = 48701 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.682\n",
            "INFO:tensorflow:loss = 0.01594323, step = 48801 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 525.306\n",
            "INFO:tensorflow:loss = 0.015855435, step = 48901 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 535.226\n",
            "INFO:tensorflow:loss = 0.015907858, step = 49001 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 444.239\n",
            "INFO:tensorflow:loss = 0.01014306, step = 49101 (0.225 sec)\n",
            "INFO:tensorflow:global_step/sec: 538.54\n",
            "INFO:tensorflow:loss = 0.0122604985, step = 49201 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 531.664\n",
            "INFO:tensorflow:loss = 0.0061300197, step = 49301 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 544.307\n",
            "INFO:tensorflow:loss = 0.009754255, step = 49401 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.184\n",
            "INFO:tensorflow:loss = 0.010754004, step = 49501 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 514.989\n",
            "INFO:tensorflow:loss = 0.015031451, step = 49601 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.478\n",
            "INFO:tensorflow:loss = 0.0052967127, step = 49701 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.697\n",
            "INFO:tensorflow:loss = 0.007391564, step = 49801 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 533.797\n",
            "INFO:tensorflow:loss = 0.011773407, step = 49901 (0.187 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 50000 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/uniform_average_ensemble_baseline/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-15:59:24\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/model.ckpt-50000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 50000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036551207, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.049044557, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.143447, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving candidate 't2_2_layer_dnn' dict for global step 50000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_2_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.037080813, average_loss/adanet/subnetwork = 0.056401946, average_loss/adanet/uniform_average_ensemble = 0.037080817, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.04785892, loss/adanet/subnetwork = 0.06400509, loss/adanet/uniform_average_ensemble = 0.047858924, prediction/mean/adanet/adanet_weighted_ensemble = 3.0937371, prediction/mean/adanet/subnetwork = 2.9943178, prediction/mean/adanet/uniform_average_ensemble = 3.0937374\n",
            "INFO:tensorflow:Saving candidate 't2_3_layer_dnn' dict for global step 50000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_3_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 3_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03804928, average_loss/adanet/subnetwork = 0.06374278, average_loss/adanet/uniform_average_ensemble = 0.03804928, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.04967243, loss/adanet/subnetwork = 0.07257131, loss/adanet/uniform_average_ensemble = 0.04967241, prediction/mean/adanet/adanet_weighted_ensemble = 3.0876217, prediction/mean/adanet/subnetwork = 2.9759707, prediction/mean/adanet/uniform_average_ensemble = 3.0876217\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-15:59:26\n",
            "INFO:tensorflow:Saving dict for global step 50000: average_loss = 0.03804928, average_loss/adanet/adanet_weighted_ensemble = 0.03804928, average_loss/adanet/subnetwork = 0.06374278, average_loss/adanet/uniform_average_ensemble = 0.03804928, global_step = 50000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.04967243, loss/adanet/adanet_weighted_ensemble = 0.04967243, loss/adanet/subnetwork = 0.07257131, loss/adanet/uniform_average_ensemble = 0.04967241, prediction/mean = 3.0876217, prediction/mean/adanet/adanet_weighted_ensemble = 3.0876217, prediction/mean/adanet/subnetwork = 2.9759707, prediction/mean/adanet/uniform_average_ensemble = 3.0876217\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 50000: /tmp/models/uniform_average_ensemble_baseline/model.ckpt-50000\n",
            "INFO:tensorflow:global_step/sec: 16.6573\n",
            "INFO:tensorflow:loss = 0.010415823, step = 50001 (6.004 sec)\n",
            "INFO:tensorflow:global_step/sec: 457.151\n",
            "INFO:tensorflow:loss = 0.0078093386, step = 50101 (0.218 sec)\n",
            "INFO:tensorflow:global_step/sec: 468.797\n",
            "INFO:tensorflow:loss = 0.009683769, step = 50201 (0.214 sec)\n",
            "INFO:tensorflow:global_step/sec: 463.878\n",
            "INFO:tensorflow:loss = 0.010256764, step = 50301 (0.215 sec)\n",
            "INFO:tensorflow:global_step/sec: 463.568\n",
            "INFO:tensorflow:loss = 0.0057664756, step = 50401 (0.217 sec)\n",
            "INFO:tensorflow:global_step/sec: 458.074\n",
            "INFO:tensorflow:loss = 0.0048193517, step = 50501 (0.217 sec)\n",
            "INFO:tensorflow:global_step/sec: 482.84\n",
            "INFO:tensorflow:loss = 0.008965603, step = 50601 (0.207 sec)\n",
            "INFO:tensorflow:global_step/sec: 454.527\n",
            "INFO:tensorflow:loss = 0.008590381, step = 50701 (0.220 sec)\n",
            "INFO:tensorflow:global_step/sec: 482.784\n",
            "INFO:tensorflow:loss = 0.010752758, step = 50801 (0.211 sec)\n",
            "INFO:tensorflow:global_step/sec: 461.979\n",
            "INFO:tensorflow:loss = 0.010941893, step = 50901 (0.213 sec)\n",
            "INFO:tensorflow:global_step/sec: 460.706\n",
            "INFO:tensorflow:loss = 0.005132697, step = 51001 (0.217 sec)\n",
            "INFO:tensorflow:global_step/sec: 459.133\n",
            "INFO:tensorflow:loss = 0.015635636, step = 51101 (0.218 sec)\n",
            "INFO:tensorflow:global_step/sec: 367.523\n",
            "INFO:tensorflow:loss = 0.007865574, step = 51201 (0.272 sec)\n",
            "INFO:tensorflow:global_step/sec: 478.043\n",
            "INFO:tensorflow:loss = 0.012698731, step = 51301 (0.210 sec)\n",
            "INFO:tensorflow:global_step/sec: 432.047\n",
            "INFO:tensorflow:loss = 0.006516525, step = 51401 (0.231 sec)\n",
            "INFO:tensorflow:global_step/sec: 465.825\n",
            "INFO:tensorflow:loss = 0.020028872, step = 51501 (0.215 sec)\n",
            "INFO:tensorflow:global_step/sec: 459.447\n",
            "INFO:tensorflow:loss = 0.008058535, step = 51601 (0.218 sec)\n",
            "INFO:tensorflow:global_step/sec: 465.614\n",
            "INFO:tensorflow:loss = 0.013191959, step = 51701 (0.215 sec)\n",
            "INFO:tensorflow:global_step/sec: 472.541\n",
            "INFO:tensorflow:loss = 0.006172248, step = 51801 (0.215 sec)\n",
            "INFO:tensorflow:global_step/sec: 461.193\n",
            "INFO:tensorflow:loss = 0.0075329277, step = 51901 (0.213 sec)\n",
            "INFO:tensorflow:global_step/sec: 468.691\n",
            "INFO:tensorflow:loss = 0.01236294, step = 52001 (0.214 sec)\n",
            "INFO:tensorflow:global_step/sec: 459.057\n",
            "INFO:tensorflow:loss = 0.009080103, step = 52101 (0.218 sec)\n",
            "INFO:tensorflow:global_step/sec: 477.911\n",
            "INFO:tensorflow:loss = 0.004784939, step = 52201 (0.209 sec)\n",
            "INFO:tensorflow:global_step/sec: 464.461\n",
            "INFO:tensorflow:loss = 0.005809378, step = 52301 (0.215 sec)\n",
            "INFO:tensorflow:global_step/sec: 467.902\n",
            "INFO:tensorflow:loss = 0.008896292, step = 52401 (0.213 sec)\n",
            "INFO:tensorflow:global_step/sec: 490.686\n",
            "INFO:tensorflow:loss = 0.012991693, step = 52501 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 485.541\n",
            "INFO:tensorflow:loss = 0.00523357, step = 52601 (0.206 sec)\n",
            "INFO:tensorflow:global_step/sec: 476.56\n",
            "INFO:tensorflow:loss = 0.0126403, step = 52701 (0.214 sec)\n",
            "INFO:tensorflow:global_step/sec: 471.963\n",
            "INFO:tensorflow:loss = 0.012190027, step = 52801 (0.207 sec)\n",
            "INFO:tensorflow:global_step/sec: 495.054\n",
            "INFO:tensorflow:loss = 0.009047846, step = 52901 (0.206 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.083\n",
            "INFO:tensorflow:loss = 0.016720409, step = 53001 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 534.56\n",
            "INFO:tensorflow:loss = 0.008708558, step = 53101 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.443\n",
            "INFO:tensorflow:loss = 0.0060455026, step = 53201 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 535.846\n",
            "INFO:tensorflow:loss = 0.0059131896, step = 53301 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.721\n",
            "INFO:tensorflow:loss = 0.008498724, step = 53401 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 533.066\n",
            "INFO:tensorflow:loss = 0.01591551, step = 53501 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.071\n",
            "INFO:tensorflow:loss = 0.007739989, step = 53601 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 444.354\n",
            "INFO:tensorflow:loss = 0.005910041, step = 53701 (0.225 sec)\n",
            "INFO:tensorflow:global_step/sec: 538.285\n",
            "INFO:tensorflow:loss = 0.008591672, step = 53801 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 542.302\n",
            "INFO:tensorflow:loss = 0.010399253, step = 53901 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.463\n",
            "INFO:tensorflow:loss = 0.0075341957, step = 54001 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 531.688\n",
            "INFO:tensorflow:loss = 0.00848561, step = 54101 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 530.457\n",
            "INFO:tensorflow:loss = 0.011938963, step = 54201 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.537\n",
            "INFO:tensorflow:loss = 0.0075793443, step = 54301 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 532.15\n",
            "INFO:tensorflow:loss = 0.007862505, step = 54401 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.441\n",
            "INFO:tensorflow:loss = 0.0062569357, step = 54501 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.834\n",
            "INFO:tensorflow:loss = 0.012258603, step = 54601 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 535.78\n",
            "INFO:tensorflow:loss = 0.008679066, step = 54701 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 529.19\n",
            "INFO:tensorflow:loss = 0.012829384, step = 54801 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 531.593\n",
            "INFO:tensorflow:loss = 0.0068805693, step = 54901 (0.185 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 55000 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/uniform_average_ensemble_baseline/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-15:59:40\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/model.ckpt-55000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 55000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036551207, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.049044557, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.143447, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving candidate 't2_2_layer_dnn' dict for global step 55000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_2_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035098273, average_loss/adanet/subnetwork = 0.04715966, average_loss/adanet/uniform_average_ensemble = 0.03509828, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.044210453, loss/adanet/subnetwork = 0.052383907, loss/adanet/uniform_average_ensemble = 0.044210464, prediction/mean/adanet/adanet_weighted_ensemble = 3.1003592, prediction/mean/adanet/subnetwork = 3.014184, prediction/mean/adanet/uniform_average_ensemble = 3.1003592\n",
            "INFO:tensorflow:Saving candidate 't2_3_layer_dnn' dict for global step 55000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_3_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 3_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03755822, average_loss/adanet/subnetwork = 0.04596351, average_loss/adanet/uniform_average_ensemble = 0.037558217, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.048664086, loss/adanet/subnetwork = 0.05456455, loss/adanet/uniform_average_ensemble = 0.04866407, prediction/mean/adanet/adanet_weighted_ensemble = 3.1223106, prediction/mean/adanet/subnetwork = 3.0800374, prediction/mean/adanet/uniform_average_ensemble = 3.1223106\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-15:59:42\n",
            "INFO:tensorflow:Saving dict for global step 55000: average_loss = 0.03755822, average_loss/adanet/adanet_weighted_ensemble = 0.03755822, average_loss/adanet/subnetwork = 0.04596351, average_loss/adanet/uniform_average_ensemble = 0.037558217, global_step = 55000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.048664086, loss/adanet/adanet_weighted_ensemble = 0.048664086, loss/adanet/subnetwork = 0.05456455, loss/adanet/uniform_average_ensemble = 0.04866407, prediction/mean = 3.1223106, prediction/mean/adanet/adanet_weighted_ensemble = 3.1223106, prediction/mean/adanet/subnetwork = 3.0800374, prediction/mean/adanet/uniform_average_ensemble = 3.1223106\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 55000: /tmp/models/uniform_average_ensemble_baseline/model.ckpt-55000\n",
            "INFO:tensorflow:global_step/sec: 16.3728\n",
            "INFO:tensorflow:loss = 0.019333825, step = 55001 (6.109 sec)\n",
            "INFO:tensorflow:global_step/sec: 534.35\n",
            "INFO:tensorflow:loss = 0.006463808, step = 55101 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.202\n",
            "INFO:tensorflow:loss = 0.007791998, step = 55201 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 521.548\n",
            "INFO:tensorflow:loss = 0.014733218, step = 55301 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 540.812\n",
            "INFO:tensorflow:loss = 0.008103497, step = 55401 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.818\n",
            "INFO:tensorflow:loss = 0.0156147145, step = 55501 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 538.834\n",
            "INFO:tensorflow:loss = 0.006833978, step = 55601 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.904\n",
            "INFO:tensorflow:loss = 0.0057001817, step = 55701 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 533.903\n",
            "INFO:tensorflow:loss = 0.013803213, step = 55801 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 422.249\n",
            "INFO:tensorflow:loss = 0.006178955, step = 55901 (0.237 sec)\n",
            "INFO:tensorflow:global_step/sec: 530.681\n",
            "INFO:tensorflow:loss = 0.010434851, step = 56001 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 527.613\n",
            "INFO:tensorflow:loss = 0.013656752, step = 56101 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.765\n",
            "INFO:tensorflow:loss = 0.007813391, step = 56201 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.927\n",
            "INFO:tensorflow:loss = 0.0053833053, step = 56301 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 546.819\n",
            "INFO:tensorflow:loss = 0.010461917, step = 56401 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 531.525\n",
            "INFO:tensorflow:loss = 0.009640503, step = 56501 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.944\n",
            "INFO:tensorflow:loss = 0.0060745412, step = 56601 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 523.056\n",
            "INFO:tensorflow:loss = 0.012264021, step = 56701 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 519.854\n",
            "INFO:tensorflow:loss = 0.008854756, step = 56801 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 529.299\n",
            "INFO:tensorflow:loss = 0.004605172, step = 56901 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 534.354\n",
            "INFO:tensorflow:loss = 0.011034519, step = 57001 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 534.068\n",
            "INFO:tensorflow:loss = 0.014240638, step = 57101 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 540.669\n",
            "INFO:tensorflow:loss = 0.006805493, step = 57201 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 528.709\n",
            "INFO:tensorflow:loss = 0.005889505, step = 57301 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 540.232\n",
            "INFO:tensorflow:loss = 0.0047724373, step = 57401 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.651\n",
            "INFO:tensorflow:loss = 0.010125456, step = 57501 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 529.984\n",
            "INFO:tensorflow:loss = 0.010492737, step = 57601 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 520.018\n",
            "INFO:tensorflow:loss = 0.0050277323, step = 57701 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 520.118\n",
            "INFO:tensorflow:loss = 0.011810246, step = 57801 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.764\n",
            "INFO:tensorflow:loss = 0.005229443, step = 57901 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.483\n",
            "INFO:tensorflow:loss = 0.010765116, step = 58001 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 540.275\n",
            "INFO:tensorflow:loss = 0.010713872, step = 58101 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.31\n",
            "INFO:tensorflow:loss = 0.008611636, step = 58201 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.61\n",
            "INFO:tensorflow:loss = 0.009485011, step = 58301 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.113\n",
            "INFO:tensorflow:loss = 0.013770765, step = 58401 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.398\n",
            "INFO:tensorflow:loss = 0.009663336, step = 58501 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 448.439\n",
            "INFO:tensorflow:loss = 0.008928772, step = 58601 (0.225 sec)\n",
            "INFO:tensorflow:global_step/sec: 529.722\n",
            "INFO:tensorflow:loss = 0.005934506, step = 58701 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.133\n",
            "INFO:tensorflow:loss = 0.00829659, step = 58801 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.184\n",
            "INFO:tensorflow:loss = 0.016658235, step = 58901 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.791\n",
            "INFO:tensorflow:loss = 0.006662611, step = 59001 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 548.952\n",
            "INFO:tensorflow:loss = 0.007583933, step = 59101 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 529.659\n",
            "INFO:tensorflow:loss = 0.0046261977, step = 59201 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 532.433\n",
            "INFO:tensorflow:loss = 0.0099191265, step = 59301 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 511.08\n",
            "INFO:tensorflow:loss = 0.004158373, step = 59401 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 546.546\n",
            "INFO:tensorflow:loss = 0.005280897, step = 59501 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 528.687\n",
            "INFO:tensorflow:loss = 0.010030862, step = 59601 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 538.318\n",
            "INFO:tensorflow:loss = 0.007858258, step = 59701 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.82\n",
            "INFO:tensorflow:loss = 0.01220257, step = 59801 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 518.055\n",
            "INFO:tensorflow:loss = 0.01195773, step = 59901 (0.193 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 60000 into /tmp/models/uniform_average_ensemble_baseline/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/uniform_average_ensemble_baseline/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-15:59:55\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/uniform_average_ensemble_baseline/model.ckpt-60000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 60000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036551207, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.049044557, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.143447, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving candidate 't2_2_layer_dnn' dict for global step 60000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_2_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.034923617, average_loss/adanet/subnetwork = 0.036563545, average_loss/adanet/uniform_average_ensemble = 0.034923624, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.04692079, loss/adanet/subnetwork = 0.049429692, loss/adanet/uniform_average_ensemble = 0.0469208, prediction/mean/adanet/adanet_weighted_ensemble = 3.1404736, prediction/mean/adanet/subnetwork = 3.1345265, prediction/mean/adanet/uniform_average_ensemble = 3.1404736\n",
            "INFO:tensorflow:Saving candidate 't2_3_layer_dnn' dict for global step 60000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_3_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 3_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.038446438, average_loss/adanet/subnetwork = 0.047027905, average_loss/adanet/uniform_average_ensemble = 0.038446438, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.05310804, loss/adanet/subnetwork = 0.0658215, loss/adanet/uniform_average_ensemble = 0.053108037, prediction/mean/adanet/adanet_weighted_ensemble = 3.1367195, prediction/mean/adanet/subnetwork = 3.1232636, prediction/mean/adanet/uniform_average_ensemble = 3.1367195\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-15:59:57\n",
            "INFO:tensorflow:Saving dict for global step 60000: average_loss = 0.038446438, average_loss/adanet/adanet_weighted_ensemble = 0.038446438, average_loss/adanet/subnetwork = 0.047027905, average_loss/adanet/uniform_average_ensemble = 0.038446438, global_step = 60000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.05310804, loss/adanet/adanet_weighted_ensemble = 0.05310804, loss/adanet/subnetwork = 0.0658215, loss/adanet/uniform_average_ensemble = 0.053108037, prediction/mean = 3.1367195, prediction/mean/adanet/adanet_weighted_ensemble = 3.1367195, prediction/mean/adanet/subnetwork = 3.1232636, prediction/mean/adanet/uniform_average_ensemble = 3.1367195\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 60000: /tmp/models/uniform_average_ensemble_baseline/model.ckpt-60000\n",
            "INFO:tensorflow:Loss for final step: 0.005916645.\n",
            "INFO:tensorflow:Finished training Adanet iteration 2\n",
            "Loss: 0.038446438\n",
            "Architecture: | 1_layer_dnn | 2_layer_dnn | 3_layer_dnn |\n"
          ]
        }
      ],
      "source": [
        "results, _ = train_and_evaluate(\"uniform_average_ensemble_baseline\")\n",
        "print(\"Loss:\", results[\"average_loss\"])\n",
        "print(\"Architecture:\", ensemble_architecture(results))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "A0vBGKeTj0Kf"
      },
      "source": [
        "These hyperparameters preduce a model that achieves **0.0393** MSE on the test\n",
        "set (exact MSE will vary depending on the hardware you're using to train the model). Notice that the ensemble is composed of 3 subnetworks, each one a hidden\n",
        "layer deeper than the previous. The most complex subnetwork is made of 3 hidden\n",
        "layers.\n",
        "\n",
        "Since `SimpleDNNGenerator` produces subnetworks of varying complexity, and our\n",
        "model gives each one an equal weight, AdaNet selected the subnetwork that most\n",
        "lowered the ensemble's training loss at each iteration, likely the one with the\n",
        "most hidden layers, since it has the most capacity, and we aren't penalizing\n",
        "more complex subnetworks (yet).\n",
        "\n",
        "Next, instead of assigning equal weight to each subnetwork, let's learn the\n",
        "mixture weights as a convex optimization problem using SGD:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 29124
        },
        "colab_type": "code",
        "id": "RXfJ6cb_KO35",
        "outputId": "330a5456-c780-49b4-c450-cf32b8264997"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': \u003ctensorflow.python.training.server_lib.ClusterSpec object at 0x7f63801de4d0\u003e, '_model_dir': '/tmp/models/learn_mixture_weights', '_protocol': None, '_save_checkpoints_steps': 5000, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': 42, '_save_summary_steps': 5000, '_device_fn': None, '_experimental_distribute': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_evaluation_master': '', '_eval_distribute': None, '_global_id_in_cluster': 0, '_master': ''}\n",
            "INFO:tensorflow:Not using Distribute Coordinator.\n",
            "INFO:tensorflow:Running training and evaluation locally (non-distributed).\n",
            "INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 5000 or save_checkpoints_secs None.\n",
            "INFO:tensorflow:Beginning training AdaNet iteration 0\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Create CheckpointSaverHook.\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving checkpoints for 0 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:loss = 5.1627307, step = 1\n",
            "INFO:tensorflow:global_step/sec: 199.628\n",
            "INFO:tensorflow:loss = 0.34482953, step = 101 (0.502 sec)\n",
            "INFO:tensorflow:global_step/sec: 626.543\n",
            "INFO:tensorflow:loss = 0.23231634, step = 201 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 613.779\n",
            "INFO:tensorflow:loss = 0.05366449, step = 301 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 626.516\n",
            "INFO:tensorflow:loss = 0.056381367, step = 401 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 625.876\n",
            "INFO:tensorflow:loss = 0.067809105, step = 501 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.434\n",
            "INFO:tensorflow:loss = 0.0430722, step = 601 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 603.879\n",
            "INFO:tensorflow:loss = 0.026319139, step = 701 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 599.869\n",
            "INFO:tensorflow:loss = 0.021756632, step = 801 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 628.871\n",
            "INFO:tensorflow:loss = 0.03447846, step = 901 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 607.309\n",
            "INFO:tensorflow:loss = 0.10228739, step = 1001 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 618.958\n",
            "INFO:tensorflow:loss = 0.03584417, step = 1101 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 610.691\n",
            "INFO:tensorflow:loss = 0.05202101, step = 1201 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 602.174\n",
            "INFO:tensorflow:loss = 0.10508361, step = 1301 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 602.13\n",
            "INFO:tensorflow:loss = 0.026265858, step = 1401 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.706\n",
            "INFO:tensorflow:loss = 0.017575853, step = 1501 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 625.614\n",
            "INFO:tensorflow:loss = 0.047641996, step = 1601 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 616.968\n",
            "INFO:tensorflow:loss = 0.04486855, step = 1701 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 625.802\n",
            "INFO:tensorflow:loss = 0.037641592, step = 1801 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 601.178\n",
            "INFO:tensorflow:loss = 0.041413702, step = 1901 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.123\n",
            "INFO:tensorflow:loss = 0.033270072, step = 2001 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 597.193\n",
            "INFO:tensorflow:loss = 0.04700596, step = 2101 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 618.954\n",
            "INFO:tensorflow:loss = 0.029409358, step = 2201 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 620.571\n",
            "INFO:tensorflow:loss = 0.034660652, step = 2301 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 480.204\n",
            "INFO:tensorflow:loss = 0.023411732, step = 2401 (0.208 sec)\n",
            "INFO:tensorflow:global_step/sec: 595.607\n",
            "INFO:tensorflow:loss = 0.07386333, step = 2501 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 616.53\n",
            "INFO:tensorflow:loss = 0.0374316, step = 2601 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.579\n",
            "INFO:tensorflow:loss = 0.059283093, step = 2701 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.598\n",
            "INFO:tensorflow:loss = 0.013467077, step = 2801 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.726\n",
            "INFO:tensorflow:loss = 0.030527176, step = 2901 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 608.521\n",
            "INFO:tensorflow:loss = 0.039863855, step = 3001 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 630.903\n",
            "INFO:tensorflow:loss = 0.028773185, step = 3101 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 609.897\n",
            "INFO:tensorflow:loss = 0.024584116, step = 3201 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 626.214\n",
            "INFO:tensorflow:loss = 0.047123127, step = 3301 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.03\n",
            "INFO:tensorflow:loss = 0.03662737, step = 3401 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 630.787\n",
            "INFO:tensorflow:loss = 0.059032194, step = 3501 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 637.877\n",
            "INFO:tensorflow:loss = 0.023170874, step = 3601 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 631.812\n",
            "INFO:tensorflow:loss = 0.015825834, step = 3701 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.521\n",
            "INFO:tensorflow:loss = 0.021646481, step = 3801 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 625.618\n",
            "INFO:tensorflow:loss = 0.019954706, step = 3901 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 625.919\n",
            "INFO:tensorflow:loss = 0.04139094, step = 4001 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.772\n",
            "INFO:tensorflow:loss = 0.019849569, step = 4101 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 628.014\n",
            "INFO:tensorflow:loss = 0.02524615, step = 4201 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.644\n",
            "INFO:tensorflow:loss = 0.013052968, step = 4301 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 563.339\n",
            "INFO:tensorflow:loss = 0.046981297, step = 4401 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 611.281\n",
            "INFO:tensorflow:loss = 0.03066413, step = 4501 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.585\n",
            "INFO:tensorflow:loss = 0.034902148, step = 4601 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 626.159\n",
            "INFO:tensorflow:loss = 0.04031996, step = 4701 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.09\n",
            "INFO:tensorflow:loss = 0.04839877, step = 4801 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 631.225\n",
            "INFO:tensorflow:loss = 0.021489367, step = 4901 (0.158 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 5000 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:00:11\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/model.ckpt-5000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_linear' dict for global step 5000: architecture/adanet/ensembles = \n",
            "W\n",
            "9adanet/iteration_0/ensemble_t0_linear/architecture/adanetB\u0010\u0008\u0007\u0012\u0000B\n",
            "| linear |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.049661, average_loss/adanet/subnetwork = 0.04963816, average_loss/adanet/uniform_average_ensemble = 0.04963816, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.063613884, loss/adanet/subnetwork = 0.06355202, loss/adanet/uniform_average_ensemble = 0.06355202, prediction/mean/adanet/adanet_weighted_ensemble = 3.1217601, prediction/mean/adanet/subnetwork = 3.1209233, prediction/mean/adanet/uniform_average_ensemble = 3.1209233\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 5000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.049852848, average_loss/adanet/subnetwork = 0.04984243, average_loss/adanet/uniform_average_ensemble = 0.04984243, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.068327785, loss/adanet/subnetwork = 0.06829582, loss/adanet/uniform_average_ensemble = 0.06829582, prediction/mean/adanet/adanet_weighted_ensemble = 3.1312768, prediction/mean/adanet/subnetwork = 3.131058, prediction/mean/adanet/uniform_average_ensemble = 3.131058\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:00:12\n",
            "INFO:tensorflow:Saving dict for global step 5000: average_loss = 0.049852848, average_loss/adanet/adanet_weighted_ensemble = 0.049852848, average_loss/adanet/subnetwork = 0.04984243, average_loss/adanet/uniform_average_ensemble = 0.04984243, global_step = 5000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.068327785, loss/adanet/adanet_weighted_ensemble = 0.068327785, loss/adanet/subnetwork = 0.06829582, loss/adanet/uniform_average_ensemble = 0.06829582, prediction/mean = 3.1312768, prediction/mean/adanet/adanet_weighted_ensemble = 3.1312768, prediction/mean/adanet/subnetwork = 3.131058, prediction/mean/adanet/uniform_average_ensemble = 3.131058\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmp/models/learn_mixture_weights/model.ckpt-5000\n",
            "INFO:tensorflow:global_step/sec: 20.1349\n",
            "INFO:tensorflow:loss = 0.013581466, step = 5001 (4.969 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.532\n",
            "INFO:tensorflow:loss = 0.019488808, step = 5101 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 605\n",
            "INFO:tensorflow:loss = 0.039483108, step = 5201 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.679\n",
            "INFO:tensorflow:loss = 0.035201266, step = 5301 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 620.86\n",
            "INFO:tensorflow:loss = 0.023337796, step = 5401 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.566\n",
            "INFO:tensorflow:loss = 0.017731346, step = 5501 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.06\n",
            "INFO:tensorflow:loss = 0.023314565, step = 5601 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 414.508\n",
            "INFO:tensorflow:loss = 0.01906262, step = 5701 (0.241 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.678\n",
            "INFO:tensorflow:loss = 0.02101638, step = 5801 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.711\n",
            "INFO:tensorflow:loss = 0.029486667, step = 5901 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 608.928\n",
            "INFO:tensorflow:loss = 0.01747806, step = 6001 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 607.283\n",
            "INFO:tensorflow:loss = 0.034387816, step = 6101 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.929\n",
            "INFO:tensorflow:loss = 0.04401755, step = 6201 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 630.418\n",
            "INFO:tensorflow:loss = 0.024472771, step = 6301 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 625.097\n",
            "INFO:tensorflow:loss = 0.034105435, step = 6401 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 626.665\n",
            "INFO:tensorflow:loss = 0.03337763, step = 6501 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.907\n",
            "INFO:tensorflow:loss = 0.039142244, step = 6601 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.999\n",
            "INFO:tensorflow:loss = 0.021583056, step = 6701 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 605.429\n",
            "INFO:tensorflow:loss = 0.028021367, step = 6801 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.894\n",
            "INFO:tensorflow:loss = 0.0100657735, step = 6901 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 628.346\n",
            "INFO:tensorflow:loss = 0.019127574, step = 7001 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.765\n",
            "INFO:tensorflow:loss = 0.03601475, step = 7101 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.799\n",
            "INFO:tensorflow:loss = 0.015298533, step = 7201 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 627.502\n",
            "INFO:tensorflow:loss = 0.041718096, step = 7301 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.547\n",
            "INFO:tensorflow:loss = 0.02148872, step = 7401 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 605.561\n",
            "INFO:tensorflow:loss = 0.016283566, step = 7501 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 607.264\n",
            "INFO:tensorflow:loss = 0.011986317, step = 7601 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.769\n",
            "INFO:tensorflow:loss = 0.024967035, step = 7701 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 608.65\n",
            "INFO:tensorflow:loss = 0.018139314, step = 7801 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 613.297\n",
            "INFO:tensorflow:loss = 0.013635318, step = 7901 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.243\n",
            "INFO:tensorflow:loss = 0.02559552, step = 8001 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.864\n",
            "INFO:tensorflow:loss = 0.02966211, step = 8101 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.909\n",
            "INFO:tensorflow:loss = 0.07569307, step = 8201 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.083\n",
            "INFO:tensorflow:loss = 0.02114915, step = 8301 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 610.975\n",
            "INFO:tensorflow:loss = 0.04814522, step = 8401 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 607.382\n",
            "INFO:tensorflow:loss = 0.017299652, step = 8501 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 627.156\n",
            "INFO:tensorflow:loss = 0.025255794, step = 8601 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 609.763\n",
            "INFO:tensorflow:loss = 0.021148201, step = 8701 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 473.761\n",
            "INFO:tensorflow:loss = 0.04819768, step = 8801 (0.211 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.501\n",
            "INFO:tensorflow:loss = 0.04284943, step = 8901 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 631.856\n",
            "INFO:tensorflow:loss = 0.04385017, step = 9001 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.595\n",
            "INFO:tensorflow:loss = 0.01906224, step = 9101 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.78\n",
            "INFO:tensorflow:loss = 0.03018801, step = 9201 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 604.095\n",
            "INFO:tensorflow:loss = 0.013036795, step = 9301 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.235\n",
            "INFO:tensorflow:loss = 0.01263145, step = 9401 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.948\n",
            "INFO:tensorflow:loss = 0.050877444, step = 9501 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.751\n",
            "INFO:tensorflow:loss = 0.022545364, step = 9601 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.383\n",
            "INFO:tensorflow:loss = 0.009367272, step = 9701 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.618\n",
            "INFO:tensorflow:loss = 0.018069556, step = 9801 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.005\n",
            "INFO:tensorflow:loss = 0.019315457, step = 9901 (0.162 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 10000 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:00:23\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/model.ckpt-10000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_linear' dict for global step 10000: architecture/adanet/ensembles = \n",
            "W\n",
            "9adanet/iteration_0/ensemble_t0_linear/architecture/adanetB\u0010\u0008\u0007\u0012\u0000B\n",
            "| linear |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.05009287, average_loss/adanet/subnetwork = 0.050082315, average_loss/adanet/uniform_average_ensemble = 0.050082315, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.06441779, loss/adanet/subnetwork = 0.06438362, loss/adanet/uniform_average_ensemble = 0.06438362, prediction/mean/adanet/adanet_weighted_ensemble = 3.1194146, prediction/mean/adanet/subnetwork = 3.118966, prediction/mean/adanet/uniform_average_ensemble = 3.118966\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 10000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.045510463, average_loss/adanet/subnetwork = 0.045428075, average_loss/adanet/uniform_average_ensemble = 0.045428075, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.054026887, loss/adanet/subnetwork = 0.054026034, loss/adanet/uniform_average_ensemble = 0.054026034, prediction/mean/adanet/adanet_weighted_ensemble = 3.0687873, prediction/mean/adanet/subnetwork = 3.069825, prediction/mean/adanet/uniform_average_ensemble = 3.069825\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:00:24\n",
            "INFO:tensorflow:Saving dict for global step 10000: average_loss = 0.045510463, average_loss/adanet/adanet_weighted_ensemble = 0.045510463, average_loss/adanet/subnetwork = 0.045428075, average_loss/adanet/uniform_average_ensemble = 0.045428075, global_step = 10000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.054026887, loss/adanet/adanet_weighted_ensemble = 0.054026887, loss/adanet/subnetwork = 0.054026034, loss/adanet/uniform_average_ensemble = 0.054026034, prediction/mean = 3.0687873, prediction/mean/adanet/adanet_weighted_ensemble = 3.0687873, prediction/mean/adanet/subnetwork = 3.069825, prediction/mean/adanet/uniform_average_ensemble = 3.069825\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 10000: /tmp/models/learn_mixture_weights/model.ckpt-10000\n",
            "INFO:tensorflow:global_step/sec: 35.2247\n",
            "INFO:tensorflow:loss = 0.021728458, step = 10001 (2.840 sec)\n",
            "INFO:tensorflow:global_step/sec: 593.43\n",
            "INFO:tensorflow:loss = 0.02273288, step = 10101 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 628.646\n",
            "INFO:tensorflow:loss = 0.031360548, step = 10201 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 478.632\n",
            "INFO:tensorflow:loss = 0.029297426, step = 10301 (0.209 sec)\n",
            "INFO:tensorflow:global_step/sec: 627.419\n",
            "INFO:tensorflow:loss = 0.022663493, step = 10401 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 625.876\n",
            "INFO:tensorflow:loss = 0.0309878, step = 10501 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 631.038\n",
            "INFO:tensorflow:loss = 0.013518688, step = 10601 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 611.44\n",
            "INFO:tensorflow:loss = 0.008507244, step = 10701 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 616.436\n",
            "INFO:tensorflow:loss = 0.016805539, step = 10801 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 615.798\n",
            "INFO:tensorflow:loss = 0.020938426, step = 10901 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 608.728\n",
            "INFO:tensorflow:loss = 0.0093169045, step = 11001 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 626.362\n",
            "INFO:tensorflow:loss = 0.02253475, step = 11101 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.185\n",
            "INFO:tensorflow:loss = 0.009709435, step = 11201 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.437\n",
            "INFO:tensorflow:loss = 0.025910603, step = 11301 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 606.911\n",
            "INFO:tensorflow:loss = 0.01590142, step = 11401 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 611.325\n",
            "INFO:tensorflow:loss = 0.056015003, step = 11501 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.223\n",
            "INFO:tensorflow:loss = 0.007529016, step = 11601 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.773\n",
            "INFO:tensorflow:loss = 0.03249779, step = 11701 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 601.446\n",
            "INFO:tensorflow:loss = 0.01331392, step = 11801 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.391\n",
            "INFO:tensorflow:loss = 0.03321218, step = 11901 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 595.454\n",
            "INFO:tensorflow:loss = 0.025778893, step = 12001 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.263\n",
            "INFO:tensorflow:loss = 0.016027156, step = 12101 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.433\n",
            "INFO:tensorflow:loss = 0.010272192, step = 12201 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 589.55\n",
            "INFO:tensorflow:loss = 0.015495548, step = 12301 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.122\n",
            "INFO:tensorflow:loss = 0.012367582, step = 12401 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 601.946\n",
            "INFO:tensorflow:loss = 0.018214615, step = 12501 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 616.158\n",
            "INFO:tensorflow:loss = 0.018225161, step = 12601 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 620.817\n",
            "INFO:tensorflow:loss = 0.020684805, step = 12701 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.306\n",
            "INFO:tensorflow:loss = 0.016095903, step = 12801 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.699\n",
            "INFO:tensorflow:loss = 0.014952449, step = 12901 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 604.737\n",
            "INFO:tensorflow:loss = 0.027632264, step = 13001 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.454\n",
            "INFO:tensorflow:loss = 0.040117525, step = 13101 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.897\n",
            "INFO:tensorflow:loss = 0.022141058, step = 13201 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 618.264\n",
            "INFO:tensorflow:loss = 0.010214046, step = 13301 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 501.741\n",
            "INFO:tensorflow:loss = 0.017293353, step = 13401 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 594.492\n",
            "INFO:tensorflow:loss = 0.019331986, step = 13501 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 564.474\n",
            "INFO:tensorflow:loss = 0.011170194, step = 13601 (0.177 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.837\n",
            "INFO:tensorflow:loss = 0.0069102095, step = 13701 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 615.847\n",
            "INFO:tensorflow:loss = 0.03940074, step = 13801 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 620.247\n",
            "INFO:tensorflow:loss = 0.014731638, step = 13901 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.781\n",
            "INFO:tensorflow:loss = 0.017168947, step = 14001 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 626.535\n",
            "INFO:tensorflow:loss = 0.015122482, step = 14101 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 631.96\n",
            "INFO:tensorflow:loss = 0.01469494, step = 14201 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 638.54\n",
            "INFO:tensorflow:loss = 0.013626468, step = 14301 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.961\n",
            "INFO:tensorflow:loss = 0.009009619, step = 14401 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 613.712\n",
            "INFO:tensorflow:loss = 0.013472855, step = 14501 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.072\n",
            "INFO:tensorflow:loss = 0.030132014, step = 14601 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 638.741\n",
            "INFO:tensorflow:loss = 0.018961016, step = 14701 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 629.933\n",
            "INFO:tensorflow:loss = 0.027708426, step = 14801 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.375\n",
            "INFO:tensorflow:loss = 0.03200151, step = 14901 (0.162 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 15000 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:00:34\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/model.ckpt-15000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_linear' dict for global step 15000: architecture/adanet/ensembles = \n",
            "W\n",
            "9adanet/iteration_0/ensemble_t0_linear/architecture/adanetB\u0010\u0008\u0007\u0012\u0000B\n",
            "| linear |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.050681442, average_loss/adanet/subnetwork = 0.05043939, average_loss/adanet/uniform_average_ensemble = 0.05043939, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.063027285, loss/adanet/subnetwork = 0.06300935, loss/adanet/uniform_average_ensemble = 0.06300935, prediction/mean/adanet/adanet_weighted_ensemble = 3.0775573, prediction/mean/adanet/subnetwork = 3.0819287, prediction/mean/adanet/uniform_average_ensemble = 3.0819287\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 15000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.05044821, average_loss/adanet/subnetwork = 0.049884323, average_loss/adanet/uniform_average_ensemble = 0.049884323, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.04868946, loss/adanet/subnetwork = 0.048252657, loss/adanet/uniform_average_ensemble = 0.048252657, prediction/mean/adanet/adanet_weighted_ensemble = 2.9932282, prediction/mean/adanet/subnetwork = 2.9956787, prediction/mean/adanet/uniform_average_ensemble = 2.9956787\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:00:35\n",
            "INFO:tensorflow:Saving dict for global step 15000: average_loss = 0.05044821, average_loss/adanet/adanet_weighted_ensemble = 0.05044821, average_loss/adanet/subnetwork = 0.049884323, average_loss/adanet/uniform_average_ensemble = 0.049884323, global_step = 15000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.04868946, loss/adanet/adanet_weighted_ensemble = 0.04868946, loss/adanet/subnetwork = 0.048252657, loss/adanet/uniform_average_ensemble = 0.048252657, prediction/mean = 2.9932282, prediction/mean/adanet/adanet_weighted_ensemble = 2.9932282, prediction/mean/adanet/subnetwork = 2.9956787, prediction/mean/adanet/uniform_average_ensemble = 2.9956787\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 15000: /tmp/models/learn_mixture_weights/model.ckpt-15000\n",
            "INFO:tensorflow:global_step/sec: 33.8276\n",
            "INFO:tensorflow:loss = 0.023542251, step = 15001 (2.957 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.885\n",
            "INFO:tensorflow:loss = 0.011751401, step = 15101 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 603.879\n",
            "INFO:tensorflow:loss = 0.03992849, step = 15201 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 639.512\n",
            "INFO:tensorflow:loss = 0.029577471, step = 15301 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 641.446\n",
            "INFO:tensorflow:loss = 0.008382542, step = 15401 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 627.093\n",
            "INFO:tensorflow:loss = 0.018933253, step = 15501 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 609.593\n",
            "INFO:tensorflow:loss = 0.021976776, step = 15601 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 625.52\n",
            "INFO:tensorflow:loss = 0.023476675, step = 15701 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 615.199\n",
            "INFO:tensorflow:loss = 0.03955758, step = 15801 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 608.006\n",
            "INFO:tensorflow:loss = 0.019894507, step = 15901 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 616.04\n",
            "INFO:tensorflow:loss = 0.009991443, step = 16001 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.223\n",
            "INFO:tensorflow:loss = 0.05179741, step = 16101 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.531\n",
            "INFO:tensorflow:loss = 0.01703541, step = 16201 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.422\n",
            "INFO:tensorflow:loss = 0.034322493, step = 16301 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 611.142\n",
            "INFO:tensorflow:loss = 0.013037438, step = 16401 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 626.511\n",
            "INFO:tensorflow:loss = 0.014361089, step = 16501 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 636.663\n",
            "INFO:tensorflow:loss = 0.012891422, step = 16601 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.676\n",
            "INFO:tensorflow:loss = 0.024711173, step = 16701 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 625.462\n",
            "INFO:tensorflow:loss = 0.01799881, step = 16801 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 634.638\n",
            "INFO:tensorflow:loss = 0.020620557, step = 16901 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.446\n",
            "INFO:tensorflow:loss = 0.01949045, step = 17001 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 616.732\n",
            "INFO:tensorflow:loss = 0.014129545, step = 17101 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 591.313\n",
            "INFO:tensorflow:loss = 0.011536442, step = 17201 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.832\n",
            "INFO:tensorflow:loss = 0.021100221, step = 17301 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 611.128\n",
            "INFO:tensorflow:loss = 0.016446067, step = 17401 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.103\n",
            "INFO:tensorflow:loss = 0.01346365, step = 17501 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.583\n",
            "INFO:tensorflow:loss = 0.013913852, step = 17601 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 606.649\n",
            "INFO:tensorflow:loss = 0.017626021, step = 17701 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.536\n",
            "INFO:tensorflow:loss = 0.023202814, step = 17801 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 448.457\n",
            "INFO:tensorflow:loss = 0.016327221, step = 17901 (0.223 sec)\n",
            "INFO:tensorflow:global_step/sec: 629.596\n",
            "INFO:tensorflow:loss = 0.019215647, step = 18001 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 618.77\n",
            "INFO:tensorflow:loss = 0.018453285, step = 18101 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.047\n",
            "INFO:tensorflow:loss = 0.017713346, step = 18201 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 615.767\n",
            "INFO:tensorflow:loss = 0.02050769, step = 18301 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.958\n",
            "INFO:tensorflow:loss = 0.0241189, step = 18401 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.095\n",
            "INFO:tensorflow:loss = 0.01636462, step = 18501 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 616.317\n",
            "INFO:tensorflow:loss = 0.015252384, step = 18601 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.237\n",
            "INFO:tensorflow:loss = 0.01345902, step = 18701 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 605.448\n",
            "INFO:tensorflow:loss = 0.009053548, step = 18801 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 620.767\n",
            "INFO:tensorflow:loss = 0.019845136, step = 18901 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 603.482\n",
            "INFO:tensorflow:loss = 0.02044722, step = 19001 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 640.454\n",
            "INFO:tensorflow:loss = 0.027293712, step = 19101 (0.156 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.329\n",
            "INFO:tensorflow:loss = 0.010133749, step = 19201 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 629.169\n",
            "INFO:tensorflow:loss = 0.023350414, step = 19301 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 625.586\n",
            "INFO:tensorflow:loss = 0.01591891, step = 19401 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.68\n",
            "INFO:tensorflow:loss = 0.014887323, step = 19501 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 602.747\n",
            "INFO:tensorflow:loss = 0.010662762, step = 19601 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 573.237\n",
            "INFO:tensorflow:loss = 0.010824166, step = 19701 (0.174 sec)\n",
            "INFO:tensorflow:global_step/sec: 598.312\n",
            "INFO:tensorflow:loss = 0.012079176, step = 19801 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 546.525\n",
            "INFO:tensorflow:loss = 0.049497757, step = 19901 (0.183 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 20000 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:00:45\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/model.ckpt-20000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_linear' dict for global step 20000: architecture/adanet/ensembles = \n",
            "W\n",
            "9adanet/iteration_0/ensemble_t0_linear/architecture/adanetB\u0010\u0008\u0007\u0012\u0000B\n",
            "| linear |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.049419947, average_loss/adanet/subnetwork = 0.049421377, average_loss/adanet/uniform_average_ensemble = 0.049421377, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.0625109, loss/adanet/subnetwork = 0.062442042, loss/adanet/uniform_average_ensemble = 0.062442042, prediction/mean/adanet/adanet_weighted_ensemble = 3.1072564, prediction/mean/adanet/subnetwork = 3.105895, prediction/mean/adanet/uniform_average_ensemble = 3.105895\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 20000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03634451, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046820804, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.1411917, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:00:46\n",
            "INFO:tensorflow:Saving dict for global step 20000: average_loss = 0.03634451, average_loss/adanet/adanet_weighted_ensemble = 0.03634451, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, global_step = 20000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.046820804, loss/adanet/adanet_weighted_ensemble = 0.046820804, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean = 3.1411917, prediction/mean/adanet/adanet_weighted_ensemble = 3.1411917, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 20000: /tmp/models/learn_mixture_weights/model.ckpt-20000\n",
            "INFO:tensorflow:Loss for final step: 0.029018745.\n",
            "INFO:tensorflow:Finished training Adanet iteration 0\n",
            "INFO:tensorflow:Beginning bookkeeping phase for iteration 0\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Starting ensemble evaluation for iteration 0\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/model.ckpt-20000\n",
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n",
            "INFO:tensorflow:Encountered end of input after 14 evaluations\n",
            "INFO:tensorflow:Computed ensemble metrics: adanet_loss/t0_linear = 0.035082, adanet_loss/t0_1_layer_dnn = 0.016848\n",
            "INFO:tensorflow:Finished ensemble evaluation for iteration 0\n",
            "INFO:tensorflow:'t0_1_layer_dnn' at index 1 is moving onto the next iteration\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Warm-starting from: (u'/tmp/models/learn_mixture_weights/model.ckpt-20000',)\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/logits/mixture_weight; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: global_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss/local_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/train_op/is_over/is_over_var_fn/is_over_var; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss/biased; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense_1/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense_1/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Overwriting checkpoint with new graph for iteration 1 to /tmp/models/learn_mixture_weights/model.ckpt-20000\n",
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n",
            "INFO:tensorflow:Finished bookkeeping phase for iteration 0\n",
            "INFO:tensorflow:Beginning training AdaNet iteration 1\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Create CheckpointSaverHook.\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/increment.ckpt-1\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving checkpoints for 20000 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:loss = 0.019054566, step = 20001\n",
            "INFO:tensorflow:global_step/sec: 147.046\n",
            "INFO:tensorflow:loss = 0.025889209, step = 20101 (0.681 sec)\n",
            "INFO:tensorflow:global_step/sec: 552.432\n",
            "INFO:tensorflow:loss = 0.0194701, step = 20201 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 551.164\n",
            "INFO:tensorflow:loss = 0.016440276, step = 20301 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.48\n",
            "INFO:tensorflow:loss = 0.021565545, step = 20401 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 534.859\n",
            "INFO:tensorflow:loss = 0.013686644, step = 20501 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.334\n",
            "INFO:tensorflow:loss = 0.01534627, step = 20601 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.697\n",
            "INFO:tensorflow:loss = 0.010481236, step = 20701 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.039\n",
            "INFO:tensorflow:loss = 0.016575025, step = 20801 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.864\n",
            "INFO:tensorflow:loss = 0.01920046, step = 20901 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 543.499\n",
            "INFO:tensorflow:loss = 0.022909012, step = 21001 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 564.064\n",
            "INFO:tensorflow:loss = 0.018595982, step = 21101 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 530.639\n",
            "INFO:tensorflow:loss = 0.019861884, step = 21201 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 493.474\n",
            "INFO:tensorflow:loss = 0.018950593, step = 21301 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 466.111\n",
            "INFO:tensorflow:loss = 0.010219138, step = 21401 (0.214 sec)\n",
            "INFO:tensorflow:global_step/sec: 488.742\n",
            "INFO:tensorflow:loss = 0.009370714, step = 21501 (0.205 sec)\n",
            "INFO:tensorflow:global_step/sec: 498.44\n",
            "INFO:tensorflow:loss = 0.008853475, step = 21601 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 497.604\n",
            "INFO:tensorflow:loss = 0.017417625, step = 21701 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 386.056\n",
            "INFO:tensorflow:loss = 0.01326526, step = 21801 (0.256 sec)\n",
            "INFO:tensorflow:global_step/sec: 454.957\n",
            "INFO:tensorflow:loss = 0.016784973, step = 21901 (0.220 sec)\n",
            "INFO:tensorflow:global_step/sec: 488.46\n",
            "INFO:tensorflow:loss = 0.01168058, step = 22001 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 482.132\n",
            "INFO:tensorflow:loss = 0.023991596, step = 22101 (0.211 sec)\n",
            "INFO:tensorflow:global_step/sec: 477.109\n",
            "INFO:tensorflow:loss = 0.015714884, step = 22201 (0.206 sec)\n",
            "INFO:tensorflow:global_step/sec: 456.932\n",
            "INFO:tensorflow:loss = 0.012404912, step = 22301 (0.219 sec)\n",
            "INFO:tensorflow:global_step/sec: 463.633\n",
            "INFO:tensorflow:loss = 0.013517365, step = 22401 (0.216 sec)\n",
            "INFO:tensorflow:global_step/sec: 454.881\n",
            "INFO:tensorflow:loss = 0.021981522, step = 22501 (0.220 sec)\n",
            "INFO:tensorflow:global_step/sec: 472.483\n",
            "INFO:tensorflow:loss = 0.018812528, step = 22601 (0.211 sec)\n",
            "INFO:tensorflow:global_step/sec: 456.809\n",
            "INFO:tensorflow:loss = 0.01767709, step = 22701 (0.219 sec)\n",
            "INFO:tensorflow:global_step/sec: 469.988\n",
            "INFO:tensorflow:loss = 0.009281049, step = 22801 (0.213 sec)\n",
            "INFO:tensorflow:global_step/sec: 462.935\n",
            "INFO:tensorflow:loss = 0.011891309, step = 22901 (0.216 sec)\n",
            "INFO:tensorflow:global_step/sec: 477.263\n",
            "INFO:tensorflow:loss = 0.013298682, step = 23001 (0.210 sec)\n",
            "INFO:tensorflow:global_step/sec: 480.495\n",
            "INFO:tensorflow:loss = 0.007244764, step = 23101 (0.208 sec)\n",
            "INFO:tensorflow:global_step/sec: 467.353\n",
            "INFO:tensorflow:loss = 0.013214206, step = 23201 (0.214 sec)\n",
            "INFO:tensorflow:global_step/sec: 473.344\n",
            "INFO:tensorflow:loss = 0.023574311, step = 23301 (0.211 sec)\n",
            "INFO:tensorflow:global_step/sec: 478.219\n",
            "INFO:tensorflow:loss = 0.019076493, step = 23401 (0.209 sec)\n",
            "INFO:tensorflow:global_step/sec: 484.055\n",
            "INFO:tensorflow:loss = 0.026565962, step = 23501 (0.207 sec)\n",
            "INFO:tensorflow:global_step/sec: 491.07\n",
            "INFO:tensorflow:loss = 0.012692229, step = 23601 (0.203 sec)\n",
            "INFO:tensorflow:global_step/sec: 473.07\n",
            "INFO:tensorflow:loss = 0.013540713, step = 23701 (0.211 sec)\n",
            "INFO:tensorflow:global_step/sec: 482.744\n",
            "INFO:tensorflow:loss = 0.01047757, step = 23801 (0.207 sec)\n",
            "INFO:tensorflow:global_step/sec: 532.071\n",
            "INFO:tensorflow:loss = 0.012808347, step = 23901 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 538.621\n",
            "INFO:tensorflow:loss = 0.014332781, step = 24001 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.836\n",
            "INFO:tensorflow:loss = 0.014820777, step = 24101 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.958\n",
            "INFO:tensorflow:loss = 0.011859844, step = 24201 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 420.766\n",
            "INFO:tensorflow:loss = 0.010935023, step = 24301 (0.238 sec)\n",
            "INFO:tensorflow:global_step/sec: 531.095\n",
            "INFO:tensorflow:loss = 0.018925918, step = 24401 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 530.31\n",
            "INFO:tensorflow:loss = 0.009583103, step = 24501 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.085\n",
            "INFO:tensorflow:loss = 0.012796979, step = 24601 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 542.179\n",
            "INFO:tensorflow:loss = 0.013194166, step = 24701 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.407\n",
            "INFO:tensorflow:loss = 0.011952517, step = 24801 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 544.704\n",
            "INFO:tensorflow:loss = 0.011510462, step = 24901 (0.184 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 25000 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:01:09\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/model.ckpt-25000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 25000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03634451, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046820804, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.1411917, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving candidate 't1_1_layer_dnn' dict for global step 25000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_1_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.038657404, average_loss/adanet/subnetwork = 0.05592768, average_loss/adanet/uniform_average_ensemble = 0.04408999, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.049713664, loss/adanet/subnetwork = 0.081614695, loss/adanet/uniform_average_ensemble = 0.06125318, prediction/mean/adanet/adanet_weighted_ensemble = 3.1109982, prediction/mean/adanet/subnetwork = 3.1655612, prediction/mean/adanet/uniform_average_ensemble = 3.15282\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 25000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03762132, average_loss/adanet/subnetwork = 0.042800613, average_loss/adanet/uniform_average_ensemble = 0.03759182, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.044555247, loss/adanet/subnetwork = 0.050511263, loss/adanet/uniform_average_ensemble = 0.04678309, prediction/mean/adanet/adanet_weighted_ensemble = 3.0817811, prediction/mean/adanet/subnetwork = 3.0765278, prediction/mean/adanet/uniform_average_ensemble = 3.1083033\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:01:12\n",
            "INFO:tensorflow:Saving dict for global step 25000: average_loss = 0.03762132, average_loss/adanet/adanet_weighted_ensemble = 0.03762132, average_loss/adanet/subnetwork = 0.042800613, average_loss/adanet/uniform_average_ensemble = 0.03759182, global_step = 25000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.044555247, loss/adanet/adanet_weighted_ensemble = 0.044555247, loss/adanet/subnetwork = 0.050511263, loss/adanet/uniform_average_ensemble = 0.04678309, prediction/mean = 3.0817811, prediction/mean/adanet/adanet_weighted_ensemble = 3.0817811, prediction/mean/adanet/subnetwork = 3.0765278, prediction/mean/adanet/uniform_average_ensemble = 3.1083033\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 25000: /tmp/models/learn_mixture_weights/model.ckpt-25000\n",
            "INFO:tensorflow:global_step/sec: 18.0384\n",
            "INFO:tensorflow:loss = 0.009066617, step = 25001 (5.545 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.72\n",
            "INFO:tensorflow:loss = 0.00968224, step = 25101 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.026\n",
            "INFO:tensorflow:loss = 0.012438122, step = 25201 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 521.134\n",
            "INFO:tensorflow:loss = 0.018114079, step = 25301 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 568.831\n",
            "INFO:tensorflow:loss = 0.009315081, step = 25401 (0.176 sec)\n",
            "INFO:tensorflow:global_step/sec: 558.529\n",
            "INFO:tensorflow:loss = 0.0072282427, step = 25501 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.526\n",
            "INFO:tensorflow:loss = 0.013896568, step = 25601 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 548.799\n",
            "INFO:tensorflow:loss = 0.013758764, step = 25701 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.94\n",
            "INFO:tensorflow:loss = 0.012665002, step = 25801 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 559.25\n",
            "INFO:tensorflow:loss = 0.009070516, step = 25901 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 564.553\n",
            "INFO:tensorflow:loss = 0.009228686, step = 26001 (0.177 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.82\n",
            "INFO:tensorflow:loss = 0.01460193, step = 26101 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 558.079\n",
            "INFO:tensorflow:loss = 0.014572715, step = 26201 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 562.503\n",
            "INFO:tensorflow:loss = 0.008317361, step = 26301 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 562.794\n",
            "INFO:tensorflow:loss = 0.013988487, step = 26401 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 552.657\n",
            "INFO:tensorflow:loss = 0.012645776, step = 26501 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 565.333\n",
            "INFO:tensorflow:loss = 0.0131017435, step = 26601 (0.177 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.727\n",
            "INFO:tensorflow:loss = 0.011169891, step = 26701 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 560.265\n",
            "INFO:tensorflow:loss = 0.013306268, step = 26801 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 369.846\n",
            "INFO:tensorflow:loss = 0.0075376267, step = 26901 (0.271 sec)\n",
            "INFO:tensorflow:global_step/sec: 566.572\n",
            "INFO:tensorflow:loss = 0.0104566505, step = 27001 (0.176 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.873\n",
            "INFO:tensorflow:loss = 0.007480262, step = 27101 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 546.972\n",
            "INFO:tensorflow:loss = 0.010309691, step = 27201 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 562.559\n",
            "INFO:tensorflow:loss = 0.016376257, step = 27301 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 568.024\n",
            "INFO:tensorflow:loss = 0.011144833, step = 27401 (0.176 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.914\n",
            "INFO:tensorflow:loss = 0.008032618, step = 27501 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.761\n",
            "INFO:tensorflow:loss = 0.00940683, step = 27601 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 555.081\n",
            "INFO:tensorflow:loss = 0.019493856, step = 27701 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.5\n",
            "INFO:tensorflow:loss = 0.0148231825, step = 27801 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 551.122\n",
            "INFO:tensorflow:loss = 0.0064866766, step = 27901 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 562.828\n",
            "INFO:tensorflow:loss = 0.0150532015, step = 28001 (0.177 sec)\n",
            "INFO:tensorflow:global_step/sec: 560.783\n",
            "INFO:tensorflow:loss = 0.016661886, step = 28101 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 552.151\n",
            "INFO:tensorflow:loss = 0.016286701, step = 28201 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 561.047\n",
            "INFO:tensorflow:loss = 0.014655964, step = 28301 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.899\n",
            "INFO:tensorflow:loss = 0.0100638345, step = 28401 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 551.408\n",
            "INFO:tensorflow:loss = 0.009675857, step = 28501 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 561.305\n",
            "INFO:tensorflow:loss = 0.01135369, step = 28601 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.214\n",
            "INFO:tensorflow:loss = 0.011211835, step = 28701 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 546.538\n",
            "INFO:tensorflow:loss = 0.01827183, step = 28801 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 561.785\n",
            "INFO:tensorflow:loss = 0.021251041, step = 28901 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 544.26\n",
            "INFO:tensorflow:loss = 0.021279141, step = 29001 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 540.196\n",
            "INFO:tensorflow:loss = 0.012094647, step = 29101 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 544.205\n",
            "INFO:tensorflow:loss = 0.014502468, step = 29201 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.549\n",
            "INFO:tensorflow:loss = 0.005638103, step = 29301 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.489\n",
            "INFO:tensorflow:loss = 0.009913086, step = 29401 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 558.341\n",
            "INFO:tensorflow:loss = 0.020633752, step = 29501 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 561.334\n",
            "INFO:tensorflow:loss = 0.008796021, step = 29601 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 488.601\n",
            "INFO:tensorflow:loss = 0.006468228, step = 29701 (0.205 sec)\n",
            "INFO:tensorflow:global_step/sec: 513.968\n",
            "INFO:tensorflow:loss = 0.013732535, step = 29801 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 571.915\n",
            "INFO:tensorflow:loss = 0.014313702, step = 29901 (0.175 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 30000 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:01:24\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/model.ckpt-30000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 30000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03634451, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046820804, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.1411917, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving candidate 't1_1_layer_dnn' dict for global step 30000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_1_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03733652, average_loss/adanet/subnetwork = 0.047193587, average_loss/adanet/uniform_average_ensemble = 0.039409403, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.044853054, loss/adanet/subnetwork = 0.0635132, loss/adanet/uniform_average_ensemble = 0.05252158, prediction/mean/adanet/adanet_weighted_ensemble = 3.0790782, prediction/mean/adanet/subnetwork = 3.0809996, prediction/mean/adanet/uniform_average_ensemble = 3.1105392\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 30000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.037665088, average_loss/adanet/subnetwork = 0.04406667, average_loss/adanet/uniform_average_ensemble = 0.035760086, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.044218667, loss/adanet/subnetwork = 0.052304745, loss/adanet/uniform_average_ensemble = 0.044707134, prediction/mean/adanet/adanet_weighted_ensemble = 3.056344, prediction/mean/adanet/subnetwork = 3.0224612, prediction/mean/adanet/uniform_average_ensemble = 3.0812702\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:01:25\n",
            "INFO:tensorflow:Saving dict for global step 30000: average_loss = 0.037665088, average_loss/adanet/adanet_weighted_ensemble = 0.037665088, average_loss/adanet/subnetwork = 0.04406667, average_loss/adanet/uniform_average_ensemble = 0.035760086, global_step = 30000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.044218667, loss/adanet/adanet_weighted_ensemble = 0.044218667, loss/adanet/subnetwork = 0.052304745, loss/adanet/uniform_average_ensemble = 0.044707134, prediction/mean = 3.056344, prediction/mean/adanet/adanet_weighted_ensemble = 3.056344, prediction/mean/adanet/subnetwork = 3.0224612, prediction/mean/adanet/uniform_average_ensemble = 3.0812702\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 30000: /tmp/models/learn_mixture_weights/model.ckpt-30000\n",
            "INFO:tensorflow:global_step/sec: 21.1545\n",
            "INFO:tensorflow:loss = 0.014034495, step = 30001 (4.731 sec)\n",
            "INFO:tensorflow:global_step/sec: 326.56\n",
            "INFO:tensorflow:loss = 0.0055237347, step = 30101 (0.303 sec)\n",
            "INFO:tensorflow:global_step/sec: 542.082\n",
            "INFO:tensorflow:loss = 0.013449315, step = 30201 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.344\n",
            "INFO:tensorflow:loss = 0.0120944735, step = 30301 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 560.215\n",
            "INFO:tensorflow:loss = 0.008284034, step = 30401 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.807\n",
            "INFO:tensorflow:loss = 0.0075836116, step = 30501 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.587\n",
            "INFO:tensorflow:loss = 0.006714839, step = 30601 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.281\n",
            "INFO:tensorflow:loss = 0.0061843027, step = 30701 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 552.92\n",
            "INFO:tensorflow:loss = 0.010956515, step = 30801 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 552.279\n",
            "INFO:tensorflow:loss = 0.019431468, step = 30901 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 543.098\n",
            "INFO:tensorflow:loss = 0.0067319167, step = 31001 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 551.082\n",
            "INFO:tensorflow:loss = 0.013767028, step = 31101 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.59\n",
            "INFO:tensorflow:loss = 0.007213892, step = 31201 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 563.71\n",
            "INFO:tensorflow:loss = 0.011032038, step = 31301 (0.177 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.354\n",
            "INFO:tensorflow:loss = 0.0062624826, step = 31401 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.183\n",
            "INFO:tensorflow:loss = 0.017503401, step = 31501 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 559.513\n",
            "INFO:tensorflow:loss = 0.008540465, step = 31601 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 559.05\n",
            "INFO:tensorflow:loss = 0.013208443, step = 31701 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.24\n",
            "INFO:tensorflow:loss = 0.0061901365, step = 31801 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 540.167\n",
            "INFO:tensorflow:loss = 0.015412205, step = 31901 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 548.237\n",
            "INFO:tensorflow:loss = 0.012029815, step = 32001 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.273\n",
            "INFO:tensorflow:loss = 0.009488794, step = 32101 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 514.282\n",
            "INFO:tensorflow:loss = 0.005871465, step = 32201 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 506.639\n",
            "INFO:tensorflow:loss = 0.008613904, step = 32301 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 506.735\n",
            "INFO:tensorflow:loss = 0.0069332877, step = 32401 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 529.669\n",
            "INFO:tensorflow:loss = 0.011869565, step = 32501 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 543.165\n",
            "INFO:tensorflow:loss = 0.009613058, step = 32601 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 480.568\n",
            "INFO:tensorflow:loss = 0.01511387, step = 32701 (0.215 sec)\n",
            "INFO:tensorflow:global_step/sec: 419.992\n",
            "INFO:tensorflow:loss = 0.013409653, step = 32801 (0.231 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.934\n",
            "INFO:tensorflow:loss = 0.011916447, step = 32901 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.913\n",
            "INFO:tensorflow:loss = 0.015191376, step = 33001 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.796\n",
            "INFO:tensorflow:loss = 0.011952508, step = 33101 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 548.781\n",
            "INFO:tensorflow:loss = 0.008621942, step = 33201 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.309\n",
            "INFO:tensorflow:loss = 0.005457497, step = 33301 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.481\n",
            "INFO:tensorflow:loss = 0.010428235, step = 33401 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.721\n",
            "INFO:tensorflow:loss = 0.010477968, step = 33501 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 555\n",
            "INFO:tensorflow:loss = 0.01792346, step = 33601 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 560.337\n",
            "INFO:tensorflow:loss = 0.00393442, step = 33701 (0.176 sec)\n",
            "INFO:tensorflow:global_step/sec: 559.703\n",
            "INFO:tensorflow:loss = 0.016329754, step = 33801 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.121\n",
            "INFO:tensorflow:loss = 0.0100037865, step = 33901 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.976\n",
            "INFO:tensorflow:loss = 0.004985449, step = 34001 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 551.858\n",
            "INFO:tensorflow:loss = 0.01096531, step = 34101 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 560.34\n",
            "INFO:tensorflow:loss = 0.009826982, step = 34201 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 559.756\n",
            "INFO:tensorflow:loss = 0.008739766, step = 34301 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.768\n",
            "INFO:tensorflow:loss = 0.0065149777, step = 34401 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 546.972\n",
            "INFO:tensorflow:loss = 0.00675286, step = 34501 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.678\n",
            "INFO:tensorflow:loss = 0.0103401635, step = 34601 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 530.867\n",
            "INFO:tensorflow:loss = 0.01190993, step = 34701 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 540.997\n",
            "INFO:tensorflow:loss = 0.016069686, step = 34801 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 555.658\n",
            "INFO:tensorflow:loss = 0.008753695, step = 34901 (0.180 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 35000 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:01:38\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/model.ckpt-35000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 35000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03634451, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046820804, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.1411917, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving candidate 't1_1_layer_dnn' dict for global step 35000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_1_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03725053, average_loss/adanet/subnetwork = 0.047201026, average_loss/adanet/uniform_average_ensemble = 0.037198033, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.042963177, loss/adanet/subnetwork = 0.054078415, loss/adanet/uniform_average_ensemble = 0.045761347, prediction/mean/adanet/adanet_weighted_ensemble = 3.0661259, prediction/mean/adanet/subnetwork = 3.0253773, prediction/mean/adanet/uniform_average_ensemble = 3.0827281\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 35000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.039036263, average_loss/adanet/subnetwork = 0.04869274, average_loss/adanet/uniform_average_ensemble = 0.0361695, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.043335173, loss/adanet/subnetwork = 0.053367704, loss/adanet/uniform_average_ensemble = 0.04201368, prediction/mean/adanet/adanet_weighted_ensemble = 3.0433319, prediction/mean/adanet/subnetwork = 3.0024989, prediction/mean/adanet/uniform_average_ensemble = 3.0712888\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:01:39\n",
            "INFO:tensorflow:Saving dict for global step 35000: average_loss = 0.039036263, average_loss/adanet/adanet_weighted_ensemble = 0.039036263, average_loss/adanet/subnetwork = 0.04869274, average_loss/adanet/uniform_average_ensemble = 0.0361695, global_step = 35000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.043335173, loss/adanet/adanet_weighted_ensemble = 0.043335173, loss/adanet/subnetwork = 0.053367704, loss/adanet/uniform_average_ensemble = 0.04201368, prediction/mean = 3.0433319, prediction/mean/adanet/adanet_weighted_ensemble = 3.0433319, prediction/mean/adanet/subnetwork = 3.0024989, prediction/mean/adanet/uniform_average_ensemble = 3.0712888\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 35000: /tmp/models/learn_mixture_weights/model.ckpt-35000\n",
            "INFO:tensorflow:global_step/sec: 22.4235\n",
            "INFO:tensorflow:loss = 0.015230963, step = 35001 (4.460 sec)\n",
            "INFO:tensorflow:global_step/sec: 542.926\n",
            "INFO:tensorflow:loss = 0.009279292, step = 35101 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 546.556\n",
            "INFO:tensorflow:loss = 0.013697773, step = 35201 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.672\n",
            "INFO:tensorflow:loss = 0.01344936, step = 35301 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 522.878\n",
            "INFO:tensorflow:loss = 0.007675332, step = 35401 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.663\n",
            "INFO:tensorflow:loss = 0.01343631, step = 35501 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 544.368\n",
            "INFO:tensorflow:loss = 0.019613128, step = 35601 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 562.942\n",
            "INFO:tensorflow:loss = 0.009834934, step = 35701 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 551.366\n",
            "INFO:tensorflow:loss = 0.012411682, step = 35801 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 552.05\n",
            "INFO:tensorflow:loss = 0.009235544, step = 35901 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 408.413\n",
            "INFO:tensorflow:loss = 0.0057943165, step = 36001 (0.245 sec)\n",
            "INFO:tensorflow:global_step/sec: 575.794\n",
            "INFO:tensorflow:loss = 0.0091551095, step = 36101 (0.174 sec)\n",
            "INFO:tensorflow:global_step/sec: 559.343\n",
            "INFO:tensorflow:loss = 0.0058207475, step = 36201 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 559.153\n",
            "INFO:tensorflow:loss = 0.0070716506, step = 36301 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 561.876\n",
            "INFO:tensorflow:loss = 0.005783346, step = 36401 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.298\n",
            "INFO:tensorflow:loss = 0.005143348, step = 36501 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 546.248\n",
            "INFO:tensorflow:loss = 0.008619576, step = 36601 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 552.245\n",
            "INFO:tensorflow:loss = 0.010861006, step = 36701 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 542.629\n",
            "INFO:tensorflow:loss = 0.007397046, step = 36801 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.169\n",
            "INFO:tensorflow:loss = 0.0064485264, step = 36901 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 552.004\n",
            "INFO:tensorflow:loss = 0.012349098, step = 37001 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.018\n",
            "INFO:tensorflow:loss = 0.011986996, step = 37101 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 555.552\n",
            "INFO:tensorflow:loss = 0.006305932, step = 37201 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 546.633\n",
            "INFO:tensorflow:loss = 0.007822029, step = 37301 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 568.047\n",
            "INFO:tensorflow:loss = 0.0048320964, step = 37401 (0.176 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.811\n",
            "INFO:tensorflow:loss = 0.008449877, step = 37501 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 559.516\n",
            "INFO:tensorflow:loss = 0.0081543, step = 37601 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.252\n",
            "INFO:tensorflow:loss = 0.009787777, step = 37701 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.838\n",
            "INFO:tensorflow:loss = 0.012568823, step = 37801 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 555.124\n",
            "INFO:tensorflow:loss = 0.008396238, step = 37901 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 562.323\n",
            "INFO:tensorflow:loss = 0.010340541, step = 38001 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 565.339\n",
            "INFO:tensorflow:loss = 0.0096965395, step = 38101 (0.177 sec)\n",
            "INFO:tensorflow:global_step/sec: 543.547\n",
            "INFO:tensorflow:loss = 0.016098175, step = 38201 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 548.606\n",
            "INFO:tensorflow:loss = 0.014968298, step = 38301 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.916\n",
            "INFO:tensorflow:loss = 0.012280576, step = 38401 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.804\n",
            "INFO:tensorflow:loss = 0.008804608, step = 38501 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 560.651\n",
            "INFO:tensorflow:loss = 0.0120964665, step = 38601 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.164\n",
            "INFO:tensorflow:loss = 0.0067613926, step = 38701 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 427.341\n",
            "INFO:tensorflow:loss = 0.0066883527, step = 38801 (0.234 sec)\n",
            "INFO:tensorflow:global_step/sec: 571.625\n",
            "INFO:tensorflow:loss = 0.012308802, step = 38901 (0.175 sec)\n",
            "INFO:tensorflow:global_step/sec: 558.974\n",
            "INFO:tensorflow:loss = 0.010688433, step = 39001 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 558.966\n",
            "INFO:tensorflow:loss = 0.009888468, step = 39101 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 554.548\n",
            "INFO:tensorflow:loss = 0.008192144, step = 39201 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 544.704\n",
            "INFO:tensorflow:loss = 0.006571155, step = 39301 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.073\n",
            "INFO:tensorflow:loss = 0.006788317, step = 39401 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.762\n",
            "INFO:tensorflow:loss = 0.00558486, step = 39501 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 567.634\n",
            "INFO:tensorflow:loss = 0.010943329, step = 39601 (0.176 sec)\n",
            "INFO:tensorflow:global_step/sec: 525.798\n",
            "INFO:tensorflow:loss = 0.007154729, step = 39701 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.283\n",
            "INFO:tensorflow:loss = 0.014514029, step = 39801 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 554.293\n",
            "INFO:tensorflow:loss = 0.015267067, step = 39901 (0.180 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 40000 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:01:51\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/model.ckpt-40000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 40000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03634451, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046820804, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.1411917, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving candidate 't1_1_layer_dnn' dict for global step 40000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_1_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035082567, average_loss/adanet/subnetwork = 0.03862485, average_loss/adanet/uniform_average_ensemble = 0.036194302, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.04224678, loss/adanet/subnetwork = 0.05337928, loss/adanet/uniform_average_ensemble = 0.048834026, prediction/mean/adanet/adanet_weighted_ensemble = 3.0901065, prediction/mean/adanet/subnetwork = 3.1667511, prediction/mean/adanet/uniform_average_ensemble = 3.153415\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 40000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035233654, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.0454976, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.111521, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:01:52\n",
            "INFO:tensorflow:Saving dict for global step 40000: average_loss = 0.035233654, average_loss/adanet/adanet_weighted_ensemble = 0.035233654, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, global_step = 40000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.0454976, loss/adanet/adanet_weighted_ensemble = 0.0454976, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean = 3.111521, prediction/mean/adanet/adanet_weighted_ensemble = 3.111521, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 40000: /tmp/models/learn_mixture_weights/model.ckpt-40000\n",
            "INFO:tensorflow:Loss for final step: 0.010157617.\n",
            "INFO:tensorflow:Finished training Adanet iteration 1\n",
            "INFO:tensorflow:Beginning bookkeeping phase for iteration 1\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Starting ensemble evaluation for iteration 1\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/model.ckpt-40000\n",
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n",
            "INFO:tensorflow:Encountered end of input after 14 evaluations\n",
            "INFO:tensorflow:Computed ensemble metrics: adanet_loss/t0_1_layer_dnn = 0.016848, adanet_loss/t1_1_layer_dnn = 0.012340, adanet_loss/t1_2_layer_dnn = 0.010463\n",
            "INFO:tensorflow:Finished ensemble evaluation for iteration 1\n",
            "INFO:tensorflow:'t1_2_layer_dnn' at index 2 is moving onto the next iteration\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Warm-starting from: (u'/tmp/models/learn_mixture_weights/model.ckpt-40000',)\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss/biased; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense_1/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense_1/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t1_2_layer_dnn/adanet_loss; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: global_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss/local_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense_1/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_0/logits/mixture_weight; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t0_1_layer_dnn/adanet_loss; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense_2/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t1_2_layer_dnn/adanet/iteration_1/candidate_t1_2_layer_dnn/adanet_loss/biased; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/logits/mixture_weight; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense_1/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/logits/mixture_weight; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t1_2_layer_dnn/adanet/iteration_1/candidate_t1_2_layer_dnn/adanet_loss/local_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t0_1_layer_dnn/adanet/iteration_1/candidate_t0_1_layer_dnn/adanet_loss/biased; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t0_1_layer_dnn/adanet/iteration_1/candidate_t0_1_layer_dnn/adanet_loss/local_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense_2/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/train_op/is_over/is_over_var_fn/is_over_var; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/train_op/is_over/is_over_var_fn/is_over_var; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Overwriting checkpoint with new graph for iteration 2 to /tmp/models/learn_mixture_weights/model.ckpt-40000\n",
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n",
            "INFO:tensorflow:Finished bookkeeping phase for iteration 1\n",
            "INFO:tensorflow:Beginning training AdaNet iteration 2\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Create CheckpointSaverHook.\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/increment.ckpt-2\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving checkpoints for 40000 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:loss = 0.012333434, step = 40001\n",
            "INFO:tensorflow:global_step/sec: 126.579\n",
            "INFO:tensorflow:loss = 0.014574185, step = 40101 (0.791 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.349\n",
            "INFO:tensorflow:loss = 0.014921842, step = 40201 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 518.006\n",
            "INFO:tensorflow:loss = 0.010299977, step = 40301 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 511.464\n",
            "INFO:tensorflow:loss = 0.012674287, step = 40401 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 507.342\n",
            "INFO:tensorflow:loss = 0.0075550666, step = 40501 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 514.007\n",
            "INFO:tensorflow:loss = 0.0071096024, step = 40601 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.326\n",
            "INFO:tensorflow:loss = 0.008376701, step = 40701 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 510.699\n",
            "INFO:tensorflow:loss = 0.01054067, step = 40801 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 510.139\n",
            "INFO:tensorflow:loss = 0.011294149, step = 40901 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.368\n",
            "INFO:tensorflow:loss = 0.01627459, step = 41001 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 508.738\n",
            "INFO:tensorflow:loss = 0.012058793, step = 41101 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 514.416\n",
            "INFO:tensorflow:loss = 0.014481242, step = 41201 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 365.343\n",
            "INFO:tensorflow:loss = 0.0084130345, step = 41301 (0.274 sec)\n",
            "INFO:tensorflow:global_step/sec: 522.469\n",
            "INFO:tensorflow:loss = 0.0060555367, step = 41401 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.109\n",
            "INFO:tensorflow:loss = 0.005273668, step = 41501 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 518.366\n",
            "INFO:tensorflow:loss = 0.0044613206, step = 41601 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 498.048\n",
            "INFO:tensorflow:loss = 0.0118937045, step = 41701 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 493.773\n",
            "INFO:tensorflow:loss = 0.009057789, step = 41801 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 513.262\n",
            "INFO:tensorflow:loss = 0.008437006, step = 41901 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 520.636\n",
            "INFO:tensorflow:loss = 0.007136828, step = 42001 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 506.639\n",
            "INFO:tensorflow:loss = 0.015660755, step = 42101 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.505\n",
            "INFO:tensorflow:loss = 0.008673275, step = 42201 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.004\n",
            "INFO:tensorflow:loss = 0.0074618477, step = 42301 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.899\n",
            "INFO:tensorflow:loss = 0.008905791, step = 42401 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 513.891\n",
            "INFO:tensorflow:loss = 0.015800748, step = 42501 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.56\n",
            "INFO:tensorflow:loss = 0.0043724724, step = 42601 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 531.669\n",
            "INFO:tensorflow:loss = 0.011617888, step = 42701 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 525.045\n",
            "INFO:tensorflow:loss = 0.0084450245, step = 42801 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 520.091\n",
            "INFO:tensorflow:loss = 0.0066436697, step = 42901 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.724\n",
            "INFO:tensorflow:loss = 0.011449961, step = 43001 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 519.241\n",
            "INFO:tensorflow:loss = 0.007268223, step = 43101 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.483\n",
            "INFO:tensorflow:loss = 0.010186626, step = 43201 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 520.181\n",
            "INFO:tensorflow:loss = 0.01616745, step = 43301 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 524.397\n",
            "INFO:tensorflow:loss = 0.011027455, step = 43401 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 521.706\n",
            "INFO:tensorflow:loss = 0.018126637, step = 43501 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 523.401\n",
            "INFO:tensorflow:loss = 0.007914587, step = 43601 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.962\n",
            "INFO:tensorflow:loss = 0.0074971775, step = 43701 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.371\n",
            "INFO:tensorflow:loss = 0.00789867, step = 43801 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 399.806\n",
            "INFO:tensorflow:loss = 0.007241773, step = 43901 (0.250 sec)\n",
            "INFO:tensorflow:global_step/sec: 522.895\n",
            "INFO:tensorflow:loss = 0.009403545, step = 44001 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.466\n",
            "INFO:tensorflow:loss = 0.0097521935, step = 44101 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 512.403\n",
            "INFO:tensorflow:loss = 0.009016202, step = 44201 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 511.554\n",
            "INFO:tensorflow:loss = 0.005688246, step = 44301 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 508.857\n",
            "INFO:tensorflow:loss = 0.011487474, step = 44401 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 519.675\n",
            "INFO:tensorflow:loss = 0.007111539, step = 44501 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.885\n",
            "INFO:tensorflow:loss = 0.0077806823, step = 44601 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.856\n",
            "INFO:tensorflow:loss = 0.0061637396, step = 44701 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 523.686\n",
            "INFO:tensorflow:loss = 0.005212597, step = 44801 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.441\n",
            "INFO:tensorflow:loss = 0.0067950566, step = 44901 (0.194 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 45000 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:02:23\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/model.ckpt-45000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 45000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035233654, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.0454976, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.111521, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving candidate 't2_2_layer_dnn' dict for global step 45000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_2_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.037295386, average_loss/adanet/subnetwork = 0.047327004, average_loss/adanet/uniform_average_ensemble = 0.038430035, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.047822557, loss/adanet/subnetwork = 0.065781765, loss/adanet/uniform_average_ensemble = 0.05305232, prediction/mean/adanet/adanet_weighted_ensemble = 3.085997, prediction/mean/adanet/subnetwork = 3.0880663, prediction/mean/adanet/uniform_average_ensemble = 3.124987\n",
            "INFO:tensorflow:Saving candidate 't2_3_layer_dnn' dict for global step 45000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_3_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 3_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036851197, average_loss/adanet/subnetwork = 0.044495832, average_loss/adanet/uniform_average_ensemble = 0.037772827, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046835795, loss/adanet/subnetwork = 0.059036292, loss/adanet/uniform_average_ensemble = 0.051192496, prediction/mean/adanet/adanet_weighted_ensemble = 3.088014, prediction/mean/adanet/subnetwork = 3.098168, prediction/mean/adanet/uniform_average_ensemble = 3.1283538\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:02:26\n",
            "INFO:tensorflow:Saving dict for global step 45000: average_loss = 0.036851197, average_loss/adanet/adanet_weighted_ensemble = 0.036851197, average_loss/adanet/subnetwork = 0.044495832, average_loss/adanet/uniform_average_ensemble = 0.037772827, global_step = 45000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.046835795, loss/adanet/adanet_weighted_ensemble = 0.046835795, loss/adanet/subnetwork = 0.059036292, loss/adanet/uniform_average_ensemble = 0.051192496, prediction/mean = 3.088014, prediction/mean/adanet/adanet_weighted_ensemble = 3.088014, prediction/mean/adanet/subnetwork = 3.098168, prediction/mean/adanet/uniform_average_ensemble = 3.1283538\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 45000: /tmp/models/learn_mixture_weights/model.ckpt-45000\n",
            "INFO:tensorflow:global_step/sec: 14.1979\n",
            "INFO:tensorflow:loss = 0.0071199634, step = 45001 (7.044 sec)\n",
            "INFO:tensorflow:global_step/sec: 497.701\n",
            "INFO:tensorflow:loss = 0.0059353793, step = 45101 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.314\n",
            "INFO:tensorflow:loss = 0.00651801, step = 45201 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.363\n",
            "INFO:tensorflow:loss = 0.015387067, step = 45301 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.689\n",
            "INFO:tensorflow:loss = 0.0054355287, step = 45401 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 511.805\n",
            "INFO:tensorflow:loss = 0.0067502894, step = 45501 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 339.793\n",
            "INFO:tensorflow:loss = 0.011912456, step = 45601 (0.294 sec)\n",
            "INFO:tensorflow:global_step/sec: 510.73\n",
            "INFO:tensorflow:loss = 0.007750011, step = 45701 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.804\n",
            "INFO:tensorflow:loss = 0.011945756, step = 45801 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 506.757\n",
            "INFO:tensorflow:loss = 0.007169023, step = 45901 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 506.547\n",
            "INFO:tensorflow:loss = 0.008177603, step = 46001 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 507.547\n",
            "INFO:tensorflow:loss = 0.01124792, step = 46101 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 512.455\n",
            "INFO:tensorflow:loss = 0.012712198, step = 46201 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 512.413\n",
            "INFO:tensorflow:loss = 0.0070413714, step = 46301 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 514.759\n",
            "INFO:tensorflow:loss = 0.009735003, step = 46401 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 513.545\n",
            "INFO:tensorflow:loss = 0.010270461, step = 46501 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 507.563\n",
            "INFO:tensorflow:loss = 0.0064955587, step = 46601 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 510.553\n",
            "INFO:tensorflow:loss = 0.00851664, step = 46701 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 514.735\n",
            "INFO:tensorflow:loss = 0.006908703, step = 46801 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 508.658\n",
            "INFO:tensorflow:loss = 0.004338132, step = 46901 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 512.271\n",
            "INFO:tensorflow:loss = 0.0055301776, step = 47001 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.886\n",
            "INFO:tensorflow:loss = 0.0064711, step = 47101 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.827\n",
            "INFO:tensorflow:loss = 0.00850641, step = 47201 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 486.474\n",
            "INFO:tensorflow:loss = 0.008801217, step = 47301 (0.205 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.668\n",
            "INFO:tensorflow:loss = 0.009333963, step = 47401 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 503.987\n",
            "INFO:tensorflow:loss = 0.0046698526, step = 47501 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 512.55\n",
            "INFO:tensorflow:loss = 0.006724437, step = 47601 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 508.251\n",
            "INFO:tensorflow:loss = 0.013553702, step = 47701 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 509.59\n",
            "INFO:tensorflow:loss = 0.012813692, step = 47801 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.239\n",
            "INFO:tensorflow:loss = 0.0055417893, step = 47901 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 450.661\n",
            "INFO:tensorflow:loss = 0.012982296, step = 48001 (0.222 sec)\n",
            "INFO:tensorflow:global_step/sec: 499.76\n",
            "INFO:tensorflow:loss = 0.012746105, step = 48101 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 373.21\n",
            "INFO:tensorflow:loss = 0.01153396, step = 48201 (0.268 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.23\n",
            "INFO:tensorflow:loss = 0.01112982, step = 48301 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 514.456\n",
            "INFO:tensorflow:loss = 0.005428352, step = 48401 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.468\n",
            "INFO:tensorflow:loss = 0.007929312, step = 48501 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 522.335\n",
            "INFO:tensorflow:loss = 0.00814957, step = 48601 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 509.149\n",
            "INFO:tensorflow:loss = 0.009238034, step = 48701 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 520.953\n",
            "INFO:tensorflow:loss = 0.0112891635, step = 48801 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 500.338\n",
            "INFO:tensorflow:loss = 0.014383599, step = 48901 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 514.332\n",
            "INFO:tensorflow:loss = 0.01564978, step = 49001 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.69\n",
            "INFO:tensorflow:loss = 0.01018079, step = 49101 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 490.16\n",
            "INFO:tensorflow:loss = 0.009571681, step = 49201 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.833\n",
            "INFO:tensorflow:loss = 0.0041006496, step = 49301 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.321\n",
            "INFO:tensorflow:loss = 0.0070481016, step = 49401 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 514.501\n",
            "INFO:tensorflow:loss = 0.0091644, step = 49501 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.832\n",
            "INFO:tensorflow:loss = 0.008436512, step = 49601 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 511.532\n",
            "INFO:tensorflow:loss = 0.0060923025, step = 49701 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.301\n",
            "INFO:tensorflow:loss = 0.00878902, step = 49801 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 518.186\n",
            "INFO:tensorflow:loss = 0.011681295, step = 49901 (0.193 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 50000 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:02:40\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/model.ckpt-50000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 50000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035233654, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.0454976, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.111521, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving candidate 't2_2_layer_dnn' dict for global step 50000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_2_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.038633004, average_loss/adanet/subnetwork = 0.056401946, average_loss/adanet/uniform_average_ensemble = 0.037080817, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.047239803, loss/adanet/subnetwork = 0.06400509, loss/adanet/uniform_average_ensemble = 0.047858924, prediction/mean/adanet/adanet_weighted_ensemble = 3.0613482, prediction/mean/adanet/subnetwork = 2.9943178, prediction/mean/adanet/uniform_average_ensemble = 3.0937374\n",
            "INFO:tensorflow:Saving candidate 't2_3_layer_dnn' dict for global step 50000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_3_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 3_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.039735794, average_loss/adanet/subnetwork = 0.06374278, average_loss/adanet/uniform_average_ensemble = 0.03804928, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.048604548, loss/adanet/subnetwork = 0.07257131, loss/adanet/uniform_average_ensemble = 0.04967241, prediction/mean/adanet/adanet_weighted_ensemble = 3.0546124, prediction/mean/adanet/subnetwork = 2.9759707, prediction/mean/adanet/uniform_average_ensemble = 3.0876217\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:02:41\n",
            "INFO:tensorflow:Saving dict for global step 50000: average_loss = 0.039735794, average_loss/adanet/adanet_weighted_ensemble = 0.039735794, average_loss/adanet/subnetwork = 0.06374278, average_loss/adanet/uniform_average_ensemble = 0.03804928, global_step = 50000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.048604548, loss/adanet/adanet_weighted_ensemble = 0.048604548, loss/adanet/subnetwork = 0.07257131, loss/adanet/uniform_average_ensemble = 0.04967241, prediction/mean = 3.0546124, prediction/mean/adanet/adanet_weighted_ensemble = 3.0546124, prediction/mean/adanet/subnetwork = 2.9759707, prediction/mean/adanet/uniform_average_ensemble = 3.0876217\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 50000: /tmp/models/learn_mixture_weights/model.ckpt-50000\n",
            "INFO:tensorflow:global_step/sec: 17.2625\n",
            "INFO:tensorflow:loss = 0.0110639315, step = 50001 (5.794 sec)\n",
            "INFO:tensorflow:global_step/sec: 501.643\n",
            "INFO:tensorflow:loss = 0.005695466, step = 50101 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 523.393\n",
            "INFO:tensorflow:loss = 0.008078868, step = 50201 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 507.372\n",
            "INFO:tensorflow:loss = 0.009924345, step = 50301 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 507.04\n",
            "INFO:tensorflow:loss = 0.007085204, step = 50401 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 359.937\n",
            "INFO:tensorflow:loss = 0.008809278, step = 50501 (0.278 sec)\n",
            "INFO:tensorflow:global_step/sec: 511.946\n",
            "INFO:tensorflow:loss = 0.0072968192, step = 50601 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 498.992\n",
            "INFO:tensorflow:loss = 0.005762454, step = 50701 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.809\n",
            "INFO:tensorflow:loss = 0.0075422693, step = 50801 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.69\n",
            "INFO:tensorflow:loss = 0.01105368, step = 50901 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 523.991\n",
            "INFO:tensorflow:loss = 0.004927204, step = 51001 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 520.224\n",
            "INFO:tensorflow:loss = 0.01071162, step = 51101 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 509.801\n",
            "INFO:tensorflow:loss = 0.0036460804, step = 51201 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 518.062\n",
            "INFO:tensorflow:loss = 0.007655938, step = 51301 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.788\n",
            "INFO:tensorflow:loss = 0.0049320585, step = 51401 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.451\n",
            "INFO:tensorflow:loss = 0.014523074, step = 51501 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 511.71\n",
            "INFO:tensorflow:loss = 0.004680957, step = 51601 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 519.23\n",
            "INFO:tensorflow:loss = 0.009958003, step = 51701 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.44\n",
            "INFO:tensorflow:loss = 0.0061810715, step = 51801 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 523.481\n",
            "INFO:tensorflow:loss = 0.009793357, step = 51901 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.061\n",
            "INFO:tensorflow:loss = 0.0073666237, step = 52001 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.477\n",
            "INFO:tensorflow:loss = 0.005454203, step = 52101 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 522.42\n",
            "INFO:tensorflow:loss = 0.0053261747, step = 52201 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 524.863\n",
            "INFO:tensorflow:loss = 0.0065975226, step = 52301 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 520.472\n",
            "INFO:tensorflow:loss = 0.005543097, step = 52401 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 513.872\n",
            "INFO:tensorflow:loss = 0.009183933, step = 52501 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 520.942\n",
            "INFO:tensorflow:loss = 0.006367198, step = 52601 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.569\n",
            "INFO:tensorflow:loss = 0.01059711, step = 52701 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 524.596\n",
            "INFO:tensorflow:loss = 0.010348743, step = 52801 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.876\n",
            "INFO:tensorflow:loss = 0.009731393, step = 52901 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 513.321\n",
            "INFO:tensorflow:loss = 0.012648952, step = 53001 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 425.103\n",
            "INFO:tensorflow:loss = 0.010847745, step = 53101 (0.237 sec)\n",
            "INFO:tensorflow:global_step/sec: 472.175\n",
            "INFO:tensorflow:loss = 0.0067536915, step = 53201 (0.211 sec)\n",
            "INFO:tensorflow:global_step/sec: 508.445\n",
            "INFO:tensorflow:loss = 0.006239564, step = 53301 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 507.511\n",
            "INFO:tensorflow:loss = 0.007061858, step = 53401 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 506.668\n",
            "INFO:tensorflow:loss = 0.008808354, step = 53501 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.237\n",
            "INFO:tensorflow:loss = 0.011044446, step = 53601 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 524.769\n",
            "INFO:tensorflow:loss = 0.004433316, step = 53701 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.609\n",
            "INFO:tensorflow:loss = 0.009089494, step = 53801 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 520.771\n",
            "INFO:tensorflow:loss = 0.00752071, step = 53901 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.569\n",
            "INFO:tensorflow:loss = 0.0043964973, step = 54001 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 518.734\n",
            "INFO:tensorflow:loss = 0.007892731, step = 54101 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 514.984\n",
            "INFO:tensorflow:loss = 0.009765172, step = 54201 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.154\n",
            "INFO:tensorflow:loss = 0.00762882, step = 54301 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 513.893\n",
            "INFO:tensorflow:loss = 0.0058656833, step = 54401 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 513.218\n",
            "INFO:tensorflow:loss = 0.004910247, step = 54501 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 521.434\n",
            "INFO:tensorflow:loss = 0.007271922, step = 54601 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 518.597\n",
            "INFO:tensorflow:loss = 0.006999065, step = 54701 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 490.141\n",
            "INFO:tensorflow:loss = 0.012415561, step = 54801 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.054\n",
            "INFO:tensorflow:loss = 0.005435602, step = 54901 (0.194 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 55000 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:02:56\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/model.ckpt-55000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 55000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035233654, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.0454976, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.111521, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving candidate 't2_2_layer_dnn' dict for global step 55000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_2_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03800079, average_loss/adanet/subnetwork = 0.04715966, average_loss/adanet/uniform_average_ensemble = 0.03509828, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.045744963, loss/adanet/subnetwork = 0.052383907, loss/adanet/uniform_average_ensemble = 0.044210464, prediction/mean/adanet/adanet_weighted_ensemble = 3.0591846, prediction/mean/adanet/subnetwork = 3.014184, prediction/mean/adanet/uniform_average_ensemble = 3.1003592\n",
            "INFO:tensorflow:Saving candidate 't2_3_layer_dnn' dict for global step 55000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_3_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 3_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.038342517, average_loss/adanet/subnetwork = 0.04596351, average_loss/adanet/uniform_average_ensemble = 0.03755822, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.04716724, loss/adanet/subnetwork = 0.05456455, loss/adanet/uniform_average_ensemble = 0.04866407, prediction/mean/adanet/adanet_weighted_ensemble = 3.076186, prediction/mean/adanet/subnetwork = 3.0800374, prediction/mean/adanet/uniform_average_ensemble = 3.1223106\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:02:57\n",
            "INFO:tensorflow:Saving dict for global step 55000: average_loss = 0.038342517, average_loss/adanet/adanet_weighted_ensemble = 0.038342517, average_loss/adanet/subnetwork = 0.04596351, average_loss/adanet/uniform_average_ensemble = 0.03755822, global_step = 55000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.04716724, loss/adanet/adanet_weighted_ensemble = 0.04716724, loss/adanet/subnetwork = 0.05456455, loss/adanet/uniform_average_ensemble = 0.04866407, prediction/mean = 3.076186, prediction/mean/adanet/adanet_weighted_ensemble = 3.076186, prediction/mean/adanet/subnetwork = 3.0800374, prediction/mean/adanet/uniform_average_ensemble = 3.1223106\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 55000: /tmp/models/learn_mixture_weights/model.ckpt-55000\n",
            "INFO:tensorflow:global_step/sec: 16.1577\n",
            "INFO:tensorflow:loss = 0.014752425, step = 55001 (6.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 494.736\n",
            "INFO:tensorflow:loss = 0.007981767, step = 55101 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 518.215\n",
            "INFO:tensorflow:loss = 0.010678162, step = 55201 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 383.055\n",
            "INFO:tensorflow:loss = 0.012075452, step = 55301 (0.261 sec)\n",
            "INFO:tensorflow:global_step/sec: 510.428\n",
            "INFO:tensorflow:loss = 0.008143797, step = 55401 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.094\n",
            "INFO:tensorflow:loss = 0.010728471, step = 55501 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.092\n",
            "INFO:tensorflow:loss = 0.009186989, step = 55601 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 510.516\n",
            "INFO:tensorflow:loss = 0.0046347613, step = 55701 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 512.997\n",
            "INFO:tensorflow:loss = 0.011230778, step = 55801 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.655\n",
            "INFO:tensorflow:loss = 0.006885918, step = 55901 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 500.41\n",
            "INFO:tensorflow:loss = 0.0068338607, step = 56001 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.938\n",
            "INFO:tensorflow:loss = 0.0075490456, step = 56101 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 500.002\n",
            "INFO:tensorflow:loss = 0.005034825, step = 56201 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 519.648\n",
            "INFO:tensorflow:loss = 0.0063436585, step = 56301 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 514.57\n",
            "INFO:tensorflow:loss = 0.006183174, step = 56401 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 511.676\n",
            "INFO:tensorflow:loss = 0.0044084843, step = 56501 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 521.58\n",
            "INFO:tensorflow:loss = 0.006386154, step = 56601 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 519.014\n",
            "INFO:tensorflow:loss = 0.0067549576, step = 56701 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.108\n",
            "INFO:tensorflow:loss = 0.0062245172, step = 56801 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.572\n",
            "INFO:tensorflow:loss = 0.005166685, step = 56901 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 508.032\n",
            "INFO:tensorflow:loss = 0.008875127, step = 57001 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 497.587\n",
            "INFO:tensorflow:loss = 0.011870863, step = 57101 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.366\n",
            "INFO:tensorflow:loss = 0.0057869563, step = 57201 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 511.401\n",
            "INFO:tensorflow:loss = 0.0070502637, step = 57301 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.202\n",
            "INFO:tensorflow:loss = 0.0072662616, step = 57401 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 512.469\n",
            "INFO:tensorflow:loss = 0.011527482, step = 57501 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 514.414\n",
            "INFO:tensorflow:loss = 0.00714942, step = 57601 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.004\n",
            "INFO:tensorflow:loss = 0.0071928333, step = 57701 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 509.157\n",
            "INFO:tensorflow:loss = 0.0080600325, step = 57801 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 394.917\n",
            "INFO:tensorflow:loss = 0.0060590906, step = 57901 (0.256 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.005\n",
            "INFO:tensorflow:loss = 0.00823138, step = 58001 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.28\n",
            "INFO:tensorflow:loss = 0.0067167096, step = 58101 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.568\n",
            "INFO:tensorflow:loss = 0.009906073, step = 58201 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 513.666\n",
            "INFO:tensorflow:loss = 0.010368018, step = 58301 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 514.403\n",
            "INFO:tensorflow:loss = 0.010448725, step = 58401 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 510.947\n",
            "INFO:tensorflow:loss = 0.0054786247, step = 58501 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 521.252\n",
            "INFO:tensorflow:loss = 0.007731803, step = 58601 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 512.93\n",
            "INFO:tensorflow:loss = 0.004775828, step = 58701 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.47\n",
            "INFO:tensorflow:loss = 0.006669793, step = 58801 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 510.186\n",
            "INFO:tensorflow:loss = 0.011582756, step = 58901 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 520.25\n",
            "INFO:tensorflow:loss = 0.006271878, step = 59001 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.256\n",
            "INFO:tensorflow:loss = 0.009062224, step = 59101 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 522.425\n",
            "INFO:tensorflow:loss = 0.0049155084, step = 59201 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 518.03\n",
            "INFO:tensorflow:loss = 0.005964897, step = 59301 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.833\n",
            "INFO:tensorflow:loss = 0.007282585, step = 59401 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.195\n",
            "INFO:tensorflow:loss = 0.0039183004, step = 59501 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 511.698\n",
            "INFO:tensorflow:loss = 0.0060636476, step = 59601 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 512.547\n",
            "INFO:tensorflow:loss = 0.0044018426, step = 59701 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.297\n",
            "INFO:tensorflow:loss = 0.0077390405, step = 59801 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 516.577\n",
            "INFO:tensorflow:loss = 0.00934967, step = 59901 (0.194 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 60000 into /tmp/models/learn_mixture_weights/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:03:11\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights/model.ckpt-60000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 60000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035233654, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.0454976, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.111521, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving candidate 't2_2_layer_dnn' dict for global step 60000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_2_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035382543, average_loss/adanet/subnetwork = 0.036563545, average_loss/adanet/uniform_average_ensemble = 0.03492362, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.04509816, loss/adanet/subnetwork = 0.049429692, loss/adanet/uniform_average_ensemble = 0.0469208, prediction/mean/adanet/adanet_weighted_ensemble = 3.083108, prediction/mean/adanet/subnetwork = 3.1345265, prediction/mean/adanet/uniform_average_ensemble = 3.1404736\n",
            "INFO:tensorflow:Saving candidate 't2_3_layer_dnn' dict for global step 60000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_3_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 3_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03865636, average_loss/adanet/subnetwork = 0.047027905, average_loss/adanet/uniform_average_ensemble = 0.038446438, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.050504602, loss/adanet/subnetwork = 0.0658215, loss/adanet/uniform_average_ensemble = 0.053108037, prediction/mean/adanet/adanet_weighted_ensemble = 3.0878685, prediction/mean/adanet/subnetwork = 3.1232636, prediction/mean/adanet/uniform_average_ensemble = 3.1367195\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:03:13\n",
            "INFO:tensorflow:Saving dict for global step 60000: average_loss = 0.03865636, average_loss/adanet/adanet_weighted_ensemble = 0.03865636, average_loss/adanet/subnetwork = 0.047027905, average_loss/adanet/uniform_average_ensemble = 0.038446438, global_step = 60000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.050504602, loss/adanet/adanet_weighted_ensemble = 0.050504602, loss/adanet/subnetwork = 0.0658215, loss/adanet/uniform_average_ensemble = 0.053108037, prediction/mean = 3.0878685, prediction/mean/adanet/adanet_weighted_ensemble = 3.0878685, prediction/mean/adanet/subnetwork = 3.1232636, prediction/mean/adanet/uniform_average_ensemble = 3.1367195\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 60000: /tmp/models/learn_mixture_weights/model.ckpt-60000\n",
            "INFO:tensorflow:Loss for final step: 0.004107115.\n",
            "INFO:tensorflow:Finished training Adanet iteration 2\n",
            "Loss: 0.03865636\n",
            "Uniform average loss: 0.038446438\n",
            "Architecture: | 1_layer_dnn | 2_layer_dnn | 3_layer_dnn |\n"
          ]
        }
      ],
      "source": [
        "#@test {\"skip\": true}\n",
        "results, _ = train_and_evaluate(\"learn_mixture_weights\", learn_mixture_weights=True)\n",
        "print(\"Loss:\", results[\"average_loss\"])\n",
        "print(\"Uniform average loss:\", results[\"average_loss/adanet/uniform_average_ensemble\"])\n",
        "print(\"Architecture:\", ensemble_architecture(results))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "BTUdJ_FWl84O"
      },
      "source": [
        "Learning the mixture weights produces a model with **0.0391** MSE, a bit better\n",
        "than the uniform average model, which the `adanet.Estimator` always compute as a\n",
        "baseline. The mixture weights were learned without regularization, so they\n",
        "risk overfitting the training set.\n",
        "\n",
        "Observe that AdaNet learned the same ensemble composition as the previous run.\n",
        "Without complexity regularization, AdaNet will favor more complex subnetworks,\n",
        "which may have worse generalization despite improving the empirical error.\n",
        "\n",
        "Finally, let's apply some **complexity regularization** by using $\\lambda \u003e 0$.\n",
        "Since this will penalize more complex subnetworks, AdaNet will select the\n",
        "candidate subnetwork that most improves the objective for its marginal\n",
        "complexity:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 29124
        },
        "colab_type": "code",
        "id": "3g_C-dT7KaTy",
        "outputId": "3e4174e3-38fc-4589-f8d8-3a9bb87b5023"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_task_type': 'worker', '_train_distribute': None, '_is_chief': True, '_cluster_spec': \u003ctensorflow.python.training.server_lib.ClusterSpec object at 0x7f637e599150\u003e, '_model_dir': '/tmp/models/learn_mixture_weights_with_complexity_regularization', '_protocol': None, '_save_checkpoints_steps': 5000, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': 42, '_save_summary_steps': 5000, '_device_fn': None, '_experimental_distribute': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_evaluation_master': '', '_eval_distribute': None, '_global_id_in_cluster': 0, '_master': ''}\n",
            "INFO:tensorflow:Not using Distribute Coordinator.\n",
            "INFO:tensorflow:Running training and evaluation locally (non-distributed).\n",
            "INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 5000 or save_checkpoints_secs None.\n",
            "INFO:tensorflow:Beginning training AdaNet iteration 0\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Create CheckpointSaverHook.\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving checkpoints for 0 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:loss = 5.1627307, step = 1\n",
            "INFO:tensorflow:global_step/sec: 188.241\n",
            "INFO:tensorflow:loss = 0.34449065, step = 101 (0.532 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.463\n",
            "INFO:tensorflow:loss = 0.23166546, step = 201 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 626.708\n",
            "INFO:tensorflow:loss = 0.053982615, step = 301 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.189\n",
            "INFO:tensorflow:loss = 0.056371797, step = 401 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 615.101\n",
            "INFO:tensorflow:loss = 0.06716307, step = 501 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.394\n",
            "INFO:tensorflow:loss = 0.042963244, step = 601 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 625.974\n",
            "INFO:tensorflow:loss = 0.026168015, step = 701 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 628.875\n",
            "INFO:tensorflow:loss = 0.021870393, step = 801 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 613.771\n",
            "INFO:tensorflow:loss = 0.034143694, step = 901 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 607.028\n",
            "INFO:tensorflow:loss = 0.10095781, step = 1001 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 608.44\n",
            "INFO:tensorflow:loss = 0.03847816, step = 1101 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 412.798\n",
            "INFO:tensorflow:loss = 0.05330515, step = 1201 (0.243 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.648\n",
            "INFO:tensorflow:loss = 0.090066634, step = 1301 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.288\n",
            "INFO:tensorflow:loss = 0.026770007, step = 1401 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.719\n",
            "INFO:tensorflow:loss = 0.037258975, step = 1501 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.245\n",
            "INFO:tensorflow:loss = 0.036077544, step = 1601 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 620.86\n",
            "INFO:tensorflow:loss = 0.034161575, step = 1701 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.446\n",
            "INFO:tensorflow:loss = 0.04626116, step = 1801 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.729\n",
            "INFO:tensorflow:loss = 0.07378493, step = 1901 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 634.373\n",
            "INFO:tensorflow:loss = 0.04918831, step = 2001 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.351\n",
            "INFO:tensorflow:loss = 0.078179196, step = 2101 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 608.447\n",
            "INFO:tensorflow:loss = 0.030299027, step = 2201 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 600.777\n",
            "INFO:tensorflow:loss = 0.024719734, step = 2301 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 637.178\n",
            "INFO:tensorflow:loss = 0.024992712, step = 2401 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 635.906\n",
            "INFO:tensorflow:loss = 0.047092065, step = 2501 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 618.769\n",
            "INFO:tensorflow:loss = 0.04721455, step = 2601 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 618.422\n",
            "INFO:tensorflow:loss = 0.038211413, step = 2701 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 606.825\n",
            "INFO:tensorflow:loss = 0.03274205, step = 2801 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 628.314\n",
            "INFO:tensorflow:loss = 0.04549656, step = 2901 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.584\n",
            "INFO:tensorflow:loss = 0.03548008, step = 3001 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.531\n",
            "INFO:tensorflow:loss = 0.024679914, step = 3101 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 587.883\n",
            "INFO:tensorflow:loss = 0.04125918, step = 3201 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 604.96\n",
            "INFO:tensorflow:loss = 0.0435674, step = 3301 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 608.988\n",
            "INFO:tensorflow:loss = 0.03460297, step = 3401 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 615.154\n",
            "INFO:tensorflow:loss = 0.06966856, step = 3501 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 611.658\n",
            "INFO:tensorflow:loss = 0.03479818, step = 3601 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 620.879\n",
            "INFO:tensorflow:loss = 0.021452513, step = 3701 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 616.964\n",
            "INFO:tensorflow:loss = 0.026122702, step = 3801 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 632.307\n",
            "INFO:tensorflow:loss = 0.031272247, step = 3901 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 618.827\n",
            "INFO:tensorflow:loss = 0.053014666, step = 4001 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.209\n",
            "INFO:tensorflow:loss = 0.028963283, step = 4101 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.817\n",
            "INFO:tensorflow:loss = 0.022142775, step = 4201 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.116\n",
            "INFO:tensorflow:loss = 0.022216441, step = 4301 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 442.413\n",
            "INFO:tensorflow:loss = 0.027055677, step = 4401 (0.226 sec)\n",
            "INFO:tensorflow:global_step/sec: 631.724\n",
            "INFO:tensorflow:loss = 0.05059754, step = 4501 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 629.501\n",
            "INFO:tensorflow:loss = 0.025971584, step = 4601 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.609\n",
            "INFO:tensorflow:loss = 0.07917491, step = 4701 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.111\n",
            "INFO:tensorflow:loss = 0.034027006, step = 4801 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 628.272\n",
            "INFO:tensorflow:loss = 0.033307493, step = 4901 (0.159 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 5000 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:03:27\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-5000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_linear' dict for global step 5000: architecture/adanet/ensembles = \n",
            "W\n",
            "9adanet/iteration_0/ensemble_t0_linear/architecture/adanetB\u0010\u0008\u0007\u0012\u0000B\n",
            "| linear |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.049661, average_loss/adanet/subnetwork = 0.04963816, average_loss/adanet/uniform_average_ensemble = 0.04963816, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.063613884, loss/adanet/subnetwork = 0.06355202, loss/adanet/uniform_average_ensemble = 0.06355202, prediction/mean/adanet/adanet_weighted_ensemble = 3.1217601, prediction/mean/adanet/subnetwork = 3.1209233, prediction/mean/adanet/uniform_average_ensemble = 3.1209233\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 5000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.04974187, average_loss/adanet/subnetwork = 0.04984243, average_loss/adanet/uniform_average_ensemble = 0.04984243, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.06797633, loss/adanet/subnetwork = 0.06829582, loss/adanet/uniform_average_ensemble = 0.06829582, prediction/mean/adanet/adanet_weighted_ensemble = 3.1288319, prediction/mean/adanet/subnetwork = 3.131058, prediction/mean/adanet/uniform_average_ensemble = 3.131058\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:03:28\n",
            "INFO:tensorflow:Saving dict for global step 5000: average_loss = 0.049661, average_loss/adanet/adanet_weighted_ensemble = 0.049661, average_loss/adanet/subnetwork = 0.04963816, average_loss/adanet/uniform_average_ensemble = 0.04963816, global_step = 5000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.063613884, loss/adanet/adanet_weighted_ensemble = 0.063613884, loss/adanet/subnetwork = 0.06355202, loss/adanet/uniform_average_ensemble = 0.06355202, prediction/mean = 3.1217601, prediction/mean/adanet/adanet_weighted_ensemble = 3.1217601, prediction/mean/adanet/subnetwork = 3.1209233, prediction/mean/adanet/uniform_average_ensemble = 3.1209233\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-5000\n",
            "INFO:tensorflow:global_step/sec: 21.8956\n",
            "INFO:tensorflow:loss = 0.026842842, step = 5001 (4.568 sec)\n",
            "INFO:tensorflow:global_step/sec: 600.298\n",
            "INFO:tensorflow:loss = 0.039310932, step = 5101 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 597.711\n",
            "INFO:tensorflow:loss = 0.030656494, step = 5201 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.437\n",
            "INFO:tensorflow:loss = 0.078128725, step = 5301 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 613.956\n",
            "INFO:tensorflow:loss = 0.021291912, step = 5401 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.026\n",
            "INFO:tensorflow:loss = 0.03251325, step = 5501 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 608.569\n",
            "INFO:tensorflow:loss = 0.028400565, step = 5601 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.545\n",
            "INFO:tensorflow:loss = 0.034857195, step = 5701 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 613.61\n",
            "INFO:tensorflow:loss = 0.037171304, step = 5801 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 615.093\n",
            "INFO:tensorflow:loss = 0.017138815, step = 5901 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 618.483\n",
            "INFO:tensorflow:loss = 0.030491468, step = 6001 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.243\n",
            "INFO:tensorflow:loss = 0.048120137, step = 6101 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 605.364\n",
            "INFO:tensorflow:loss = 0.044583086, step = 6201 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.665\n",
            "INFO:tensorflow:loss = 0.04749332, step = 6301 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 616.094\n",
            "INFO:tensorflow:loss = 0.07128419, step = 6401 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 618.628\n",
            "INFO:tensorflow:loss = 0.03341365, step = 6501 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 615.128\n",
            "INFO:tensorflow:loss = 0.019353844, step = 6601 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 606.77\n",
            "INFO:tensorflow:loss = 0.03313767, step = 6701 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 634.739\n",
            "INFO:tensorflow:loss = 0.021211505, step = 6801 (0.157 sec)\n",
            "INFO:tensorflow:global_step/sec: 609.775\n",
            "INFO:tensorflow:loss = 0.018065577, step = 6901 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.386\n",
            "INFO:tensorflow:loss = 0.031387277, step = 7001 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 629.906\n",
            "INFO:tensorflow:loss = 0.032881733, step = 7101 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.16\n",
            "INFO:tensorflow:loss = 0.01538456, step = 7201 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 552.77\n",
            "INFO:tensorflow:loss = 0.041718945, step = 7301 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 611.15\n",
            "INFO:tensorflow:loss = 0.037094295, step = 7401 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 607.566\n",
            "INFO:tensorflow:loss = 0.016327433, step = 7501 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.613\n",
            "INFO:tensorflow:loss = 0.0201954, step = 7601 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.235\n",
            "INFO:tensorflow:loss = 0.027472034, step = 7701 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 613.527\n",
            "INFO:tensorflow:loss = 0.032032184, step = 7801 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 585.669\n",
            "INFO:tensorflow:loss = 0.013514966, step = 7901 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 429.863\n",
            "INFO:tensorflow:loss = 0.03239342, step = 8001 (0.232 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.344\n",
            "INFO:tensorflow:loss = 0.027077636, step = 8101 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.207\n",
            "INFO:tensorflow:loss = 0.0360922, step = 8201 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.215\n",
            "INFO:tensorflow:loss = 0.03275392, step = 8301 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.83\n",
            "INFO:tensorflow:loss = 0.051111933, step = 8401 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 580.464\n",
            "INFO:tensorflow:loss = 0.017032994, step = 8501 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.715\n",
            "INFO:tensorflow:loss = 0.06303735, step = 8601 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 616.724\n",
            "INFO:tensorflow:loss = 0.022497727, step = 8701 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.413\n",
            "INFO:tensorflow:loss = 0.042914927, step = 8801 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 609.559\n",
            "INFO:tensorflow:loss = 0.042905785, step = 8901 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 627.4\n",
            "INFO:tensorflow:loss = 0.04343162, step = 9001 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.756\n",
            "INFO:tensorflow:loss = 0.030193526, step = 9101 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 628.196\n",
            "INFO:tensorflow:loss = 0.030161744, step = 9201 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 634.051\n",
            "INFO:tensorflow:loss = 0.028942654, step = 9301 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 594.902\n",
            "INFO:tensorflow:loss = 0.01270803, step = 9401 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 620.798\n",
            "INFO:tensorflow:loss = 0.051534586, step = 9501 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 613.203\n",
            "INFO:tensorflow:loss = 0.022428801, step = 9601 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 628.789\n",
            "INFO:tensorflow:loss = 0.0095187435, step = 9701 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 623.873\n",
            "INFO:tensorflow:loss = 0.031217653, step = 9801 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 631.536\n",
            "INFO:tensorflow:loss = 0.028995795, step = 9901 (0.158 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 10000 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:03:39\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-10000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_linear' dict for global step 10000: architecture/adanet/ensembles = \n",
            "W\n",
            "9adanet/iteration_0/ensemble_t0_linear/architecture/adanetB\u0010\u0008\u0007\u0012\u0000B\n",
            "| linear |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.05009287, average_loss/adanet/subnetwork = 0.050082315, average_loss/adanet/uniform_average_ensemble = 0.050082315, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.06441779, loss/adanet/subnetwork = 0.06438362, loss/adanet/uniform_average_ensemble = 0.06438362, prediction/mean/adanet/adanet_weighted_ensemble = 3.1194146, prediction/mean/adanet/subnetwork = 3.118966, prediction/mean/adanet/uniform_average_ensemble = 3.118966\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 10000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.045708533, average_loss/adanet/subnetwork = 0.045428075, average_loss/adanet/uniform_average_ensemble = 0.045428075, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.054037668, loss/adanet/subnetwork = 0.054026034, loss/adanet/uniform_average_ensemble = 0.054026034, prediction/mean/adanet/adanet_weighted_ensemble = 3.0663967, prediction/mean/adanet/subnetwork = 3.069825, prediction/mean/adanet/uniform_average_ensemble = 3.069825\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:03:40\n",
            "INFO:tensorflow:Saving dict for global step 10000: average_loss = 0.045708533, average_loss/adanet/adanet_weighted_ensemble = 0.045708533, average_loss/adanet/subnetwork = 0.045428075, average_loss/adanet/uniform_average_ensemble = 0.045428075, global_step = 10000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.054037668, loss/adanet/adanet_weighted_ensemble = 0.054037668, loss/adanet/subnetwork = 0.054026034, loss/adanet/uniform_average_ensemble = 0.054026034, prediction/mean = 3.0663967, prediction/mean/adanet/adanet_weighted_ensemble = 3.0663967, prediction/mean/adanet/subnetwork = 3.069825, prediction/mean/adanet/uniform_average_ensemble = 3.069825\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 10000: /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-10000\n",
            "INFO:tensorflow:global_step/sec: 32.2254\n",
            "INFO:tensorflow:loss = 0.021720316, step = 10001 (3.104 sec)\n",
            "INFO:tensorflow:global_step/sec: 607.589\n",
            "INFO:tensorflow:loss = 0.030225167, step = 10101 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 610.277\n",
            "INFO:tensorflow:loss = 0.03127557, step = 10201 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 585.793\n",
            "INFO:tensorflow:loss = 0.029002503, step = 10301 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.619\n",
            "INFO:tensorflow:loss = 0.023051744, step = 10401 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 615.73\n",
            "INFO:tensorflow:loss = 0.031586163, step = 10501 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 630.624\n",
            "INFO:tensorflow:loss = 0.013289812, step = 10601 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 620.036\n",
            "INFO:tensorflow:loss = 0.008458757, step = 10701 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 613.512\n",
            "INFO:tensorflow:loss = 0.028491888, step = 10801 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.529\n",
            "INFO:tensorflow:loss = 0.02119983, step = 10901 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.607\n",
            "INFO:tensorflow:loss = 0.0094689485, step = 11001 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 610.81\n",
            "INFO:tensorflow:loss = 0.038508512, step = 11101 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 616.146\n",
            "INFO:tensorflow:loss = 0.009698494, step = 11201 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 609.749\n",
            "INFO:tensorflow:loss = 0.054243505, step = 11301 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 626.414\n",
            "INFO:tensorflow:loss = 0.03519901, step = 11401 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.685\n",
            "INFO:tensorflow:loss = 0.049500115, step = 11501 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.127\n",
            "INFO:tensorflow:loss = 0.007494917, step = 11601 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 585.312\n",
            "INFO:tensorflow:loss = 0.03195551, step = 11701 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 616.857\n",
            "INFO:tensorflow:loss = 0.013583911, step = 11801 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 610.486\n",
            "INFO:tensorflow:loss = 0.047530938, step = 11901 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.658\n",
            "INFO:tensorflow:loss = 0.025300248, step = 12001 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.409\n",
            "INFO:tensorflow:loss = 0.017053518, step = 12101 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 586.627\n",
            "INFO:tensorflow:loss = 0.010506482, step = 12201 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.549\n",
            "INFO:tensorflow:loss = 0.015713342, step = 12301 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 410.146\n",
            "INFO:tensorflow:loss = 0.01227211, step = 12401 (0.249 sec)\n",
            "INFO:tensorflow:global_step/sec: 606.02\n",
            "INFO:tensorflow:loss = 0.018147415, step = 12501 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 613.418\n",
            "INFO:tensorflow:loss = 0.018431192, step = 12601 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 608.954\n",
            "INFO:tensorflow:loss = 0.020963538, step = 12701 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 624.309\n",
            "INFO:tensorflow:loss = 0.01593941, step = 12801 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 613.96\n",
            "INFO:tensorflow:loss = 0.036629334, step = 12901 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.236\n",
            "INFO:tensorflow:loss = 0.04917693, step = 13001 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 589.202\n",
            "INFO:tensorflow:loss = 0.035332013, step = 13101 (0.170 sec)\n",
            "INFO:tensorflow:global_step/sec: 626.378\n",
            "INFO:tensorflow:loss = 0.030816792, step = 13201 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 608.032\n",
            "INFO:tensorflow:loss = 0.010110773, step = 13301 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.678\n",
            "INFO:tensorflow:loss = 0.017346883, step = 13401 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 618.227\n",
            "INFO:tensorflow:loss = 0.01919973, step = 13501 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.873\n",
            "INFO:tensorflow:loss = 0.011193079, step = 13601 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 613.203\n",
            "INFO:tensorflow:loss = 0.0067360615, step = 13701 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 597.965\n",
            "INFO:tensorflow:loss = 0.039885808, step = 13801 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 602.017\n",
            "INFO:tensorflow:loss = 0.014925557, step = 13901 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.682\n",
            "INFO:tensorflow:loss = 0.016707938, step = 14001 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.643\n",
            "INFO:tensorflow:loss = 0.015013415, step = 14101 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 606.911\n",
            "INFO:tensorflow:loss = 0.014811715, step = 14201 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 629.941\n",
            "INFO:tensorflow:loss = 0.013648225, step = 14301 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.862\n",
            "INFO:tensorflow:loss = 0.009129133, step = 14401 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 610.311\n",
            "INFO:tensorflow:loss = 0.013653677, step = 14501 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 607.028\n",
            "INFO:tensorflow:loss = 0.030663429, step = 14601 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 609.983\n",
            "INFO:tensorflow:loss = 0.018810896, step = 14701 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 606.947\n",
            "INFO:tensorflow:loss = 0.027355086, step = 14801 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 595.702\n",
            "INFO:tensorflow:loss = 0.031680383, step = 14901 (0.168 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 15000 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:03:50\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-15000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_linear' dict for global step 15000: architecture/adanet/ensembles = \n",
            "W\n",
            "9adanet/iteration_0/ensemble_t0_linear/architecture/adanetB\u0010\u0008\u0007\u0012\u0000B\n",
            "| linear |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.050681442, average_loss/adanet/subnetwork = 0.05043939, average_loss/adanet/uniform_average_ensemble = 0.05043939, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.063027285, loss/adanet/subnetwork = 0.06300935, loss/adanet/uniform_average_ensemble = 0.06300935, prediction/mean/adanet/adanet_weighted_ensemble = 3.0775573, prediction/mean/adanet/subnetwork = 3.0819287, prediction/mean/adanet/uniform_average_ensemble = 3.0819287\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 15000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.050998196, average_loss/adanet/subnetwork = 0.049884323, average_loss/adanet/uniform_average_ensemble = 0.049884323, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.049118735, loss/adanet/subnetwork = 0.048252657, loss/adanet/uniform_average_ensemble = 0.048252657, prediction/mean/adanet/adanet_weighted_ensemble = 2.990887, prediction/mean/adanet/subnetwork = 2.9956787, prediction/mean/adanet/uniform_average_ensemble = 2.9956787\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:03:51\n",
            "INFO:tensorflow:Saving dict for global step 15000: average_loss = 0.050998196, average_loss/adanet/adanet_weighted_ensemble = 0.050998196, average_loss/adanet/subnetwork = 0.049884323, average_loss/adanet/uniform_average_ensemble = 0.049884323, global_step = 15000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.049118735, loss/adanet/adanet_weighted_ensemble = 0.049118735, loss/adanet/subnetwork = 0.048252657, loss/adanet/uniform_average_ensemble = 0.048252657, prediction/mean = 2.990887, prediction/mean/adanet/adanet_weighted_ensemble = 2.990887, prediction/mean/adanet/subnetwork = 2.9956787, prediction/mean/adanet/uniform_average_ensemble = 2.9956787\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 15000: /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-15000\n",
            "INFO:tensorflow:global_step/sec: 32.5682\n",
            "INFO:tensorflow:loss = 0.023777971, step = 15001 (3.071 sec)\n",
            "INFO:tensorflow:global_step/sec: 607.29\n",
            "INFO:tensorflow:loss = 0.011859877, step = 15101 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 611.4\n",
            "INFO:tensorflow:loss = 0.025546592, step = 15201 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.245\n",
            "INFO:tensorflow:loss = 0.02911094, step = 15301 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 607.194\n",
            "INFO:tensorflow:loss = 0.008381752, step = 15401 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 604.993\n",
            "INFO:tensorflow:loss = 0.018735591, step = 15501 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 597.183\n",
            "INFO:tensorflow:loss = 0.022403702, step = 15601 (0.168 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.5\n",
            "INFO:tensorflow:loss = 0.023885053, step = 15701 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.855\n",
            "INFO:tensorflow:loss = 0.05157975, step = 15801 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.332\n",
            "INFO:tensorflow:loss = 0.02016547, step = 15901 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.346\n",
            "INFO:tensorflow:loss = 0.01000012, step = 16001 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.62\n",
            "INFO:tensorflow:loss = 0.050864153, step = 16101 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.475\n",
            "INFO:tensorflow:loss = 0.016598087, step = 16201 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 615.093\n",
            "INFO:tensorflow:loss = 0.034943447, step = 16301 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 608.751\n",
            "INFO:tensorflow:loss = 0.013129476, step = 16401 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 598.895\n",
            "INFO:tensorflow:loss = 0.013929198, step = 16501 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 587.064\n",
            "INFO:tensorflow:loss = 0.0130764805, step = 16601 (0.171 sec)\n",
            "INFO:tensorflow:global_step/sec: 618.334\n",
            "INFO:tensorflow:loss = 0.024172928, step = 16701 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 607.234\n",
            "INFO:tensorflow:loss = 0.01767306, step = 16801 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 422.605\n",
            "INFO:tensorflow:loss = 0.020104785, step = 16901 (0.237 sec)\n",
            "INFO:tensorflow:global_step/sec: 629.592\n",
            "INFO:tensorflow:loss = 0.01963634, step = 17001 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 563.539\n",
            "INFO:tensorflow:loss = 0.014243526, step = 17101 (0.177 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.635\n",
            "INFO:tensorflow:loss = 0.011520493, step = 17201 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 494.471\n",
            "INFO:tensorflow:loss = 0.020737208, step = 17301 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.199\n",
            "INFO:tensorflow:loss = 0.0168445, step = 17401 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 608.132\n",
            "INFO:tensorflow:loss = 0.013647025, step = 17501 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 572.577\n",
            "INFO:tensorflow:loss = 0.013766556, step = 17601 (0.175 sec)\n",
            "INFO:tensorflow:global_step/sec: 577.047\n",
            "INFO:tensorflow:loss = 0.017335687, step = 17701 (0.173 sec)\n",
            "INFO:tensorflow:global_step/sec: 621.373\n",
            "INFO:tensorflow:loss = 0.029417565, step = 17801 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 618.8\n",
            "INFO:tensorflow:loss = 0.016654348, step = 17901 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 619.29\n",
            "INFO:tensorflow:loss = 0.019140113, step = 18001 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 618.927\n",
            "INFO:tensorflow:loss = 0.018077476, step = 18101 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 608.831\n",
            "INFO:tensorflow:loss = 0.017867178, step = 18201 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 601.067\n",
            "INFO:tensorflow:loss = 0.020510823, step = 18301 (0.166 sec)\n",
            "INFO:tensorflow:global_step/sec: 597.557\n",
            "INFO:tensorflow:loss = 0.024087986, step = 18401 (0.167 sec)\n",
            "INFO:tensorflow:global_step/sec: 622.467\n",
            "INFO:tensorflow:loss = 0.016705524, step = 18501 (0.161 sec)\n",
            "INFO:tensorflow:global_step/sec: 615.411\n",
            "INFO:tensorflow:loss = 0.015320602, step = 18601 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 627.798\n",
            "INFO:tensorflow:loss = 0.013655036, step = 18701 (0.159 sec)\n",
            "INFO:tensorflow:global_step/sec: 605.261\n",
            "INFO:tensorflow:loss = 0.009015914, step = 18801 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 614.866\n",
            "INFO:tensorflow:loss = 0.019498402, step = 18901 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 609.503\n",
            "INFO:tensorflow:loss = 0.019944657, step = 19001 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 606.237\n",
            "INFO:tensorflow:loss = 0.027836736, step = 19101 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 612.888\n",
            "INFO:tensorflow:loss = 0.010139037, step = 19201 (0.163 sec)\n",
            "INFO:tensorflow:global_step/sec: 632.543\n",
            "INFO:tensorflow:loss = 0.022753734, step = 19301 (0.158 sec)\n",
            "INFO:tensorflow:global_step/sec: 609.37\n",
            "INFO:tensorflow:loss = 0.016327728, step = 19401 (0.164 sec)\n",
            "INFO:tensorflow:global_step/sec: 603.777\n",
            "INFO:tensorflow:loss = 0.01446814, step = 19501 (0.165 sec)\n",
            "INFO:tensorflow:global_step/sec: 617.529\n",
            "INFO:tensorflow:loss = 0.010882783, step = 19601 (0.162 sec)\n",
            "INFO:tensorflow:global_step/sec: 602.773\n",
            "INFO:tensorflow:loss = 0.01085731, step = 19701 (0.169 sec)\n",
            "INFO:tensorflow:global_step/sec: 613.049\n",
            "INFO:tensorflow:loss = 0.012006026, step = 19801 (0.160 sec)\n",
            "INFO:tensorflow:global_step/sec: 408.268\n",
            "INFO:tensorflow:loss = 0.048679292, step = 19901 (0.245 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 20000 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:04:02\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-20000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_linear' dict for global step 20000: architecture/adanet/ensembles = \n",
            "W\n",
            "9adanet/iteration_0/ensemble_t0_linear/architecture/adanetB\u0010\u0008\u0007\u0012\u0000B\n",
            "| linear |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.049419947, average_loss/adanet/subnetwork = 0.049421377, average_loss/adanet/uniform_average_ensemble = 0.049421377, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.0625109, loss/adanet/subnetwork = 0.062442042, loss/adanet/uniform_average_ensemble = 0.062442042, prediction/mean/adanet/adanet_weighted_ensemble = 3.1072564, prediction/mean/adanet/subnetwork = 3.105895, prediction/mean/adanet/uniform_average_ensemble = 3.105895\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 20000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036191724, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046480708, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.1387463, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:04:03\n",
            "INFO:tensorflow:Saving dict for global step 20000: average_loss = 0.036191724, average_loss/adanet/adanet_weighted_ensemble = 0.036191724, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, global_step = 20000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.046480708, loss/adanet/adanet_weighted_ensemble = 0.046480708, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean = 3.1387463, prediction/mean/adanet/adanet_weighted_ensemble = 3.1387463, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 20000: /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-20000\n",
            "INFO:tensorflow:Loss for final step: 0.02949296.\n",
            "INFO:tensorflow:Finished training Adanet iteration 0\n",
            "INFO:tensorflow:Beginning bookkeeping phase for iteration 0\n",
            "INFO:tensorflow:Building iteration 0\n",
            "INFO:tensorflow:Building subnetwork 'linear'\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Starting ensemble evaluation for iteration 0\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-20000\n",
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n",
            "INFO:tensorflow:Encountered end of input after 14 evaluations\n",
            "INFO:tensorflow:Computed ensemble metrics: adanet_loss/t0_linear = 0.035082, adanet_loss/t0_1_layer_dnn = 0.031534\n",
            "INFO:tensorflow:Finished ensemble evaluation for iteration 0\n",
            "INFO:tensorflow:'t0_1_layer_dnn' at index 1 is moving onto the next iteration\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights_with_complexity_regularization/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Warm-starting from: (u'/tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-20000',)\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/logits/mixture_weight; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: global_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss/local_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/train_op/is_over/is_over_var_fn/is_over_var; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss/biased; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense_1/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense_1/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Overwriting checkpoint with new graph for iteration 1 to /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-20000\n",
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n",
            "INFO:tensorflow:Finished bookkeeping phase for iteration 0\n",
            "INFO:tensorflow:Beginning training AdaNet iteration 1\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights_with_complexity_regularization/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Create CheckpointSaverHook.\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/increment.ckpt-1\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving checkpoints for 20000 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:loss = 0.018656276, step = 20001\n",
            "INFO:tensorflow:global_step/sec: 126.47\n",
            "INFO:tensorflow:loss = 0.025519662, step = 20101 (0.792 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.848\n",
            "INFO:tensorflow:loss = 0.019204883, step = 20201 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.297\n",
            "INFO:tensorflow:loss = 0.016156942, step = 20301 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 353.071\n",
            "INFO:tensorflow:loss = 0.021228356, step = 20401 (0.283 sec)\n",
            "INFO:tensorflow:global_step/sec: 540.042\n",
            "INFO:tensorflow:loss = 0.013405463, step = 20501 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.427\n",
            "INFO:tensorflow:loss = 0.015034497, step = 20601 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 528.983\n",
            "INFO:tensorflow:loss = 0.010285204, step = 20701 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.793\n",
            "INFO:tensorflow:loss = 0.016196731, step = 20801 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 554.207\n",
            "INFO:tensorflow:loss = 0.018870529, step = 20901 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.788\n",
            "INFO:tensorflow:loss = 0.02250357, step = 21001 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 534.257\n",
            "INFO:tensorflow:loss = 0.018208357, step = 21101 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.666\n",
            "INFO:tensorflow:loss = 0.019593507, step = 21201 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.909\n",
            "INFO:tensorflow:loss = 0.018875318, step = 21301 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.783\n",
            "INFO:tensorflow:loss = 0.012622515, step = 21401 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 538.645\n",
            "INFO:tensorflow:loss = 0.009199008, step = 21501 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 552.389\n",
            "INFO:tensorflow:loss = 0.008601103, step = 21601 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 542.005\n",
            "INFO:tensorflow:loss = 0.017047679, step = 21701 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.188\n",
            "INFO:tensorflow:loss = 0.015726648, step = 21801 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.702\n",
            "INFO:tensorflow:loss = 0.016526222, step = 21901 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 542.532\n",
            "INFO:tensorflow:loss = 0.011439471, step = 22001 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.032\n",
            "INFO:tensorflow:loss = 0.025854949, step = 22101 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.398\n",
            "INFO:tensorflow:loss = 0.017800728, step = 22201 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.409\n",
            "INFO:tensorflow:loss = 0.012128357, step = 22301 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 554.986\n",
            "INFO:tensorflow:loss = 0.013639286, step = 22401 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 548.634\n",
            "INFO:tensorflow:loss = 0.02203528, step = 22501 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 540.742\n",
            "INFO:tensorflow:loss = 0.01220283, step = 22601 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.442\n",
            "INFO:tensorflow:loss = 0.019015614, step = 22701 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.327\n",
            "INFO:tensorflow:loss = 0.013728116, step = 22801 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.968\n",
            "INFO:tensorflow:loss = 0.012520758, step = 22901 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 563.949\n",
            "INFO:tensorflow:loss = 0.015283684, step = 23001 (0.177 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.858\n",
            "INFO:tensorflow:loss = 0.01151904, step = 23101 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 384.903\n",
            "INFO:tensorflow:loss = 0.018598802, step = 23201 (0.257 sec)\n",
            "INFO:tensorflow:global_step/sec: 554.351\n",
            "INFO:tensorflow:loss = 0.025367094, step = 23301 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.655\n",
            "INFO:tensorflow:loss = 0.016584951, step = 23401 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 528.611\n",
            "INFO:tensorflow:loss = 0.02993238, step = 23501 (0.189 sec)\n",
            "INFO:tensorflow:global_step/sec: 555.118\n",
            "INFO:tensorflow:loss = 0.013529665, step = 23601 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 544.416\n",
            "INFO:tensorflow:loss = 0.012705052, step = 23701 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 554.068\n",
            "INFO:tensorflow:loss = 0.012524294, step = 23801 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.253\n",
            "INFO:tensorflow:loss = 0.010380931, step = 23901 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.243\n",
            "INFO:tensorflow:loss = 0.019286085, step = 24001 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 546.765\n",
            "INFO:tensorflow:loss = 0.014981048, step = 24101 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 524.271\n",
            "INFO:tensorflow:loss = 0.01222017, step = 24201 (0.191 sec)\n",
            "INFO:tensorflow:global_step/sec: 438.285\n",
            "INFO:tensorflow:loss = 0.014058795, step = 24301 (0.228 sec)\n",
            "INFO:tensorflow:global_step/sec: 520.719\n",
            "INFO:tensorflow:loss = 0.016617503, step = 24401 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 548.462\n",
            "INFO:tensorflow:loss = 0.009422993, step = 24501 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 546.362\n",
            "INFO:tensorflow:loss = 0.012403663, step = 24601 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 534.216\n",
            "INFO:tensorflow:loss = 0.013287688, step = 24701 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.075\n",
            "INFO:tensorflow:loss = 0.011229211, step = 24801 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 555.741\n",
            "INFO:tensorflow:loss = 0.011336029, step = 24901 (0.180 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 25000 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights_with_complexity_regularization/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:04:26\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-25000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 25000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036191724, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046480708, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.1387463, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving candidate 't1_1_layer_dnn' dict for global step 25000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_1_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.038604483, average_loss/adanet/subnetwork = 0.05592768, average_loss/adanet/uniform_average_ensemble = 0.04408999, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.04941739, loss/adanet/subnetwork = 0.081614695, loss/adanet/uniform_average_ensemble = 0.06125318, prediction/mean/adanet/adanet_weighted_ensemble = 3.10835, prediction/mean/adanet/subnetwork = 3.1655612, prediction/mean/adanet/uniform_average_ensemble = 3.15282\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 25000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03742792, average_loss/adanet/subnetwork = 0.042800613, average_loss/adanet/uniform_average_ensemble = 0.03759182, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.04411319, loss/adanet/subnetwork = 0.050511263, loss/adanet/uniform_average_ensemble = 0.04678309, prediction/mean/adanet/adanet_weighted_ensemble = 3.0799336, prediction/mean/adanet/subnetwork = 3.0765278, prediction/mean/adanet/uniform_average_ensemble = 3.1083033\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:04:29\n",
            "INFO:tensorflow:Saving dict for global step 25000: average_loss = 0.038604483, average_loss/adanet/adanet_weighted_ensemble = 0.038604483, average_loss/adanet/subnetwork = 0.05592768, average_loss/adanet/uniform_average_ensemble = 0.04408999, global_step = 25000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.04941739, loss/adanet/adanet_weighted_ensemble = 0.04941739, loss/adanet/subnetwork = 0.081614695, loss/adanet/uniform_average_ensemble = 0.06125318, prediction/mean = 3.10835, prediction/mean/adanet/adanet_weighted_ensemble = 3.10835, prediction/mean/adanet/subnetwork = 3.1655612, prediction/mean/adanet/uniform_average_ensemble = 3.15282\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 25000: /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-25000\n",
            "INFO:tensorflow:global_step/sec: 17.8793\n",
            "INFO:tensorflow:loss = 0.0087926425, step = 25001 (5.596 sec)\n",
            "INFO:tensorflow:global_step/sec: 533.808\n",
            "INFO:tensorflow:loss = 0.008187551, step = 25101 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.468\n",
            "INFO:tensorflow:loss = 0.01260198, step = 25201 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.105\n",
            "INFO:tensorflow:loss = 0.018600041, step = 25301 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.609\n",
            "INFO:tensorflow:loss = 0.009139011, step = 25401 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 551.021\n",
            "INFO:tensorflow:loss = 0.007311764, step = 25501 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.103\n",
            "INFO:tensorflow:loss = 0.0143518625, step = 25601 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 538.321\n",
            "INFO:tensorflow:loss = 0.0137332855, step = 25701 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 344.69\n",
            "INFO:tensorflow:loss = 0.012674173, step = 25801 (0.290 sec)\n",
            "INFO:tensorflow:global_step/sec: 543.718\n",
            "INFO:tensorflow:loss = 0.009046188, step = 25901 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 542.72\n",
            "INFO:tensorflow:loss = 0.0092414785, step = 26001 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 543.295\n",
            "INFO:tensorflow:loss = 0.014648741, step = 26101 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.858\n",
            "INFO:tensorflow:loss = 0.014480362, step = 26201 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 555.43\n",
            "INFO:tensorflow:loss = 0.009010304, step = 26301 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 546.466\n",
            "INFO:tensorflow:loss = 0.014557325, step = 26401 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.001\n",
            "INFO:tensorflow:loss = 0.018106863, step = 26501 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 546.977\n",
            "INFO:tensorflow:loss = 0.011541239, step = 26601 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.694\n",
            "INFO:tensorflow:loss = 0.011146317, step = 26701 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.297\n",
            "INFO:tensorflow:loss = 0.012705506, step = 26801 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 517.344\n",
            "INFO:tensorflow:loss = 0.007095962, step = 26901 (0.193 sec)\n",
            "INFO:tensorflow:global_step/sec: 486.44\n",
            "INFO:tensorflow:loss = 0.010517583, step = 27001 (0.207 sec)\n",
            "INFO:tensorflow:global_step/sec: 499.428\n",
            "INFO:tensorflow:loss = 0.0072074467, step = 27101 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 476.367\n",
            "INFO:tensorflow:loss = 0.0104720015, step = 27201 (0.210 sec)\n",
            "INFO:tensorflow:global_step/sec: 484.799\n",
            "INFO:tensorflow:loss = 0.01702473, step = 27301 (0.206 sec)\n",
            "INFO:tensorflow:global_step/sec: 511.658\n",
            "INFO:tensorflow:loss = 0.010749651, step = 27401 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 438.281\n",
            "INFO:tensorflow:loss = 0.0083185015, step = 27501 (0.228 sec)\n",
            "INFO:tensorflow:global_step/sec: 491.121\n",
            "INFO:tensorflow:loss = 0.00985959, step = 27601 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 489.929\n",
            "INFO:tensorflow:loss = 0.019051343, step = 27701 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.237\n",
            "INFO:tensorflow:loss = 0.016090725, step = 27801 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 499.079\n",
            "INFO:tensorflow:loss = 0.0069275717, step = 27901 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 485.814\n",
            "INFO:tensorflow:loss = 0.015142027, step = 28001 (0.207 sec)\n",
            "INFO:tensorflow:global_step/sec: 471.643\n",
            "INFO:tensorflow:loss = 0.01681554, step = 28101 (0.211 sec)\n",
            "INFO:tensorflow:global_step/sec: 499.423\n",
            "INFO:tensorflow:loss = 0.017564762, step = 28201 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 497.456\n",
            "INFO:tensorflow:loss = 0.015120167, step = 28301 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 356.943\n",
            "INFO:tensorflow:loss = 0.01060288, step = 28401 (0.280 sec)\n",
            "INFO:tensorflow:global_step/sec: 490.249\n",
            "INFO:tensorflow:loss = 0.009240018, step = 28501 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 462.524\n",
            "INFO:tensorflow:loss = 0.012949839, step = 28601 (0.216 sec)\n",
            "INFO:tensorflow:global_step/sec: 506.029\n",
            "INFO:tensorflow:loss = 0.012082284, step = 28701 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 508.613\n",
            "INFO:tensorflow:loss = 0.01845267, step = 28801 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 498.251\n",
            "INFO:tensorflow:loss = 0.02236254, step = 28901 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.004\n",
            "INFO:tensorflow:loss = 0.022868171, step = 29001 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 495.703\n",
            "INFO:tensorflow:loss = 0.011914679, step = 29101 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 501.965\n",
            "INFO:tensorflow:loss = 0.015929814, step = 29201 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 491.947\n",
            "INFO:tensorflow:loss = 0.0059978645, step = 29301 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 489.872\n",
            "INFO:tensorflow:loss = 0.009584345, step = 29401 (0.206 sec)\n",
            "INFO:tensorflow:global_step/sec: 503.923\n",
            "INFO:tensorflow:loss = 0.021724503, step = 29501 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 483.685\n",
            "INFO:tensorflow:loss = 0.00967145, step = 29601 (0.207 sec)\n",
            "INFO:tensorflow:global_step/sec: 472.103\n",
            "INFO:tensorflow:loss = 0.0070765545, step = 29701 (0.212 sec)\n",
            "INFO:tensorflow:global_step/sec: 491.183\n",
            "INFO:tensorflow:loss = 0.013255347, step = 29801 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 494.122\n",
            "INFO:tensorflow:loss = 0.015302105, step = 29901 (0.202 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 30000 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights_with_complexity_regularization/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:04:42\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-30000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 30000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036191724, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046480708, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.1387463, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving candidate 't1_1_layer_dnn' dict for global step 30000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_1_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.037431348, average_loss/adanet/subnetwork = 0.047193587, average_loss/adanet/uniform_average_ensemble = 0.039409403, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.044711642, loss/adanet/subnetwork = 0.0635132, loss/adanet/uniform_average_ensemble = 0.05252158, prediction/mean/adanet/adanet_weighted_ensemble = 3.076714, prediction/mean/adanet/subnetwork = 3.0809996, prediction/mean/adanet/uniform_average_ensemble = 3.1105392\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 30000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.037218932, average_loss/adanet/subnetwork = 0.04406667, average_loss/adanet/uniform_average_ensemble = 0.035760086, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.043378487, loss/adanet/subnetwork = 0.052304745, loss/adanet/uniform_average_ensemble = 0.044707134, prediction/mean/adanet/adanet_weighted_ensemble = 3.0610068, prediction/mean/adanet/subnetwork = 3.0224612, prediction/mean/adanet/uniform_average_ensemble = 3.0812702\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:04:44\n",
            "INFO:tensorflow:Saving dict for global step 30000: average_loss = 0.037218932, average_loss/adanet/adanet_weighted_ensemble = 0.037218932, average_loss/adanet/subnetwork = 0.04406667, average_loss/adanet/uniform_average_ensemble = 0.035760086, global_step = 30000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.043378487, loss/adanet/adanet_weighted_ensemble = 0.043378487, loss/adanet/subnetwork = 0.052304745, loss/adanet/uniform_average_ensemble = 0.044707134, prediction/mean = 3.0610068, prediction/mean/adanet/adanet_weighted_ensemble = 3.0610068, prediction/mean/adanet/subnetwork = 3.0224612, prediction/mean/adanet/uniform_average_ensemble = 3.0812702\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 30000: /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-30000\n",
            "INFO:tensorflow:global_step/sec: 19.7501\n",
            "INFO:tensorflow:loss = 0.014787567, step = 30001 (5.064 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.84\n",
            "INFO:tensorflow:loss = 0.0060889265, step = 30101 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.385\n",
            "INFO:tensorflow:loss = 0.014240649, step = 30201 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 544.748\n",
            "INFO:tensorflow:loss = 0.0132650975, step = 30301 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 538.045\n",
            "INFO:tensorflow:loss = 0.009037494, step = 30401 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.084\n",
            "INFO:tensorflow:loss = 0.008346892, step = 30501 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 544.491\n",
            "INFO:tensorflow:loss = 0.006911318, step = 30601 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 555.213\n",
            "INFO:tensorflow:loss = 0.006248268, step = 30701 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.194\n",
            "INFO:tensorflow:loss = 0.011634795, step = 30801 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 531.118\n",
            "INFO:tensorflow:loss = 0.017662454, step = 30901 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 524.422\n",
            "INFO:tensorflow:loss = 0.007159453, step = 31001 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 440.587\n",
            "INFO:tensorflow:loss = 0.013616986, step = 31101 (0.234 sec)\n",
            "INFO:tensorflow:global_step/sec: 397.999\n",
            "INFO:tensorflow:loss = 0.0074774446, step = 31201 (0.244 sec)\n",
            "INFO:tensorflow:global_step/sec: 540.05\n",
            "INFO:tensorflow:loss = 0.0125078745, step = 31301 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 525.091\n",
            "INFO:tensorflow:loss = 0.0062818495, step = 31401 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 538.543\n",
            "INFO:tensorflow:loss = 0.01724846, step = 31501 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.421\n",
            "INFO:tensorflow:loss = 0.008689038, step = 31601 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 534.493\n",
            "INFO:tensorflow:loss = 0.013662458, step = 31701 (0.187 sec)\n",
            "INFO:tensorflow:global_step/sec: 508.295\n",
            "INFO:tensorflow:loss = 0.0071566887, step = 31801 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.403\n",
            "INFO:tensorflow:loss = 0.015569165, step = 31901 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.296\n",
            "INFO:tensorflow:loss = 0.011912314, step = 32001 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 548.547\n",
            "INFO:tensorflow:loss = 0.009985223, step = 32101 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.458\n",
            "INFO:tensorflow:loss = 0.005538259, step = 32201 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.804\n",
            "INFO:tensorflow:loss = 0.008909384, step = 32301 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 554.868\n",
            "INFO:tensorflow:loss = 0.006963819, step = 32401 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.046\n",
            "INFO:tensorflow:loss = 0.012406438, step = 32501 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 560.913\n",
            "INFO:tensorflow:loss = 0.010020049, step = 32601 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.961\n",
            "INFO:tensorflow:loss = 0.015626423, step = 32701 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 538.709\n",
            "INFO:tensorflow:loss = 0.013246004, step = 32801 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.109\n",
            "INFO:tensorflow:loss = 0.012512751, step = 32901 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 543.948\n",
            "INFO:tensorflow:loss = 0.016440626, step = 33001 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 559.597\n",
            "INFO:tensorflow:loss = 0.011751309, step = 33101 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 548.859\n",
            "INFO:tensorflow:loss = 0.0085565215, step = 33201 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.856\n",
            "INFO:tensorflow:loss = 0.0062072994, step = 33301 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.567\n",
            "INFO:tensorflow:loss = 0.01206644, step = 33401 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 555.954\n",
            "INFO:tensorflow:loss = 0.0120011205, step = 33501 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 542.217\n",
            "INFO:tensorflow:loss = 0.015643965, step = 33601 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.647\n",
            "INFO:tensorflow:loss = 0.00461392, step = 33701 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 554.05\n",
            "INFO:tensorflow:loss = 0.018861912, step = 33801 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 367.602\n",
            "INFO:tensorflow:loss = 0.010566024, step = 33901 (0.272 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.153\n",
            "INFO:tensorflow:loss = 0.0054750834, step = 34001 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 543.919\n",
            "INFO:tensorflow:loss = 0.011503474, step = 34101 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 540.409\n",
            "INFO:tensorflow:loss = 0.010721846, step = 34201 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.282\n",
            "INFO:tensorflow:loss = 0.00977681, step = 34301 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 540.24\n",
            "INFO:tensorflow:loss = 0.006933369, step = 34401 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.544\n",
            "INFO:tensorflow:loss = 0.008018806, step = 34501 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 551.353\n",
            "INFO:tensorflow:loss = 0.010969609, step = 34601 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 551.046\n",
            "INFO:tensorflow:loss = 0.013125555, step = 34701 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.381\n",
            "INFO:tensorflow:loss = 0.018013261, step = 34801 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 539.424\n",
            "INFO:tensorflow:loss = 0.012451946, step = 34901 (0.185 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 35000 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights_with_complexity_regularization/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:04:56\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-35000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 35000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036191724, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046480708, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.1387463, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving candidate 't1_1_layer_dnn' dict for global step 35000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_1_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.037397612, average_loss/adanet/subnetwork = 0.047201026, average_loss/adanet/uniform_average_ensemble = 0.037198033, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.042938285, loss/adanet/subnetwork = 0.054078415, loss/adanet/uniform_average_ensemble = 0.045761347, prediction/mean/adanet/adanet_weighted_ensemble = 3.064317, prediction/mean/adanet/subnetwork = 3.0253773, prediction/mean/adanet/uniform_average_ensemble = 3.0827281\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 35000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.037893172, average_loss/adanet/subnetwork = 0.04869274, average_loss/adanet/uniform_average_ensemble = 0.0361695, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.04240233, loss/adanet/subnetwork = 0.053367704, loss/adanet/uniform_average_ensemble = 0.04201368, prediction/mean/adanet/adanet_weighted_ensemble = 3.0527809, prediction/mean/adanet/subnetwork = 3.0024989, prediction/mean/adanet/uniform_average_ensemble = 3.0712888\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:04:57\n",
            "INFO:tensorflow:Saving dict for global step 35000: average_loss = 0.037893172, average_loss/adanet/adanet_weighted_ensemble = 0.037893172, average_loss/adanet/subnetwork = 0.04869274, average_loss/adanet/uniform_average_ensemble = 0.0361695, global_step = 35000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.04240233, loss/adanet/adanet_weighted_ensemble = 0.04240233, loss/adanet/subnetwork = 0.053367704, loss/adanet/uniform_average_ensemble = 0.04201368, prediction/mean = 3.0527809, prediction/mean/adanet/adanet_weighted_ensemble = 3.0527809, prediction/mean/adanet/subnetwork = 3.0024989, prediction/mean/adanet/uniform_average_ensemble = 3.0712888\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 35000: /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-35000\n",
            "INFO:tensorflow:global_step/sec: 22.4821\n",
            "INFO:tensorflow:loss = 0.016108558, step = 35001 (4.450 sec)\n",
            "INFO:tensorflow:global_step/sec: 554.367\n",
            "INFO:tensorflow:loss = 0.009662176, step = 35101 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.167\n",
            "INFO:tensorflow:loss = 0.013212465, step = 35201 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.436\n",
            "INFO:tensorflow:loss = 0.015238007, step = 35301 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.866\n",
            "INFO:tensorflow:loss = 0.0082730735, step = 35401 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 561.712\n",
            "INFO:tensorflow:loss = 0.014469749, step = 35501 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 562.379\n",
            "INFO:tensorflow:loss = 0.018691026, step = 35601 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.746\n",
            "INFO:tensorflow:loss = 0.010978206, step = 35701 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 551.867\n",
            "INFO:tensorflow:loss = 0.012770843, step = 35801 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.304\n",
            "INFO:tensorflow:loss = 0.010215035, step = 35901 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 551.453\n",
            "INFO:tensorflow:loss = 0.0070546973, step = 36001 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 560.481\n",
            "INFO:tensorflow:loss = 0.009964033, step = 36101 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 551.688\n",
            "INFO:tensorflow:loss = 0.0064386544, step = 36201 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.872\n",
            "INFO:tensorflow:loss = 0.009141, step = 36301 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.744\n",
            "INFO:tensorflow:loss = 0.0070760474, step = 36401 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.492\n",
            "INFO:tensorflow:loss = 0.006196186, step = 36501 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.715\n",
            "INFO:tensorflow:loss = 0.009804886, step = 36601 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.397\n",
            "INFO:tensorflow:loss = 0.012993045, step = 36701 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.611\n",
            "INFO:tensorflow:loss = 0.008648639, step = 36801 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 552.343\n",
            "INFO:tensorflow:loss = 0.0065253368, step = 36901 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.33\n",
            "INFO:tensorflow:loss = 0.013697723, step = 37001 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 537.675\n",
            "INFO:tensorflow:loss = 0.0126334345, step = 37101 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 355.054\n",
            "INFO:tensorflow:loss = 0.007980874, step = 37201 (0.282 sec)\n",
            "INFO:tensorflow:global_step/sec: 526.596\n",
            "INFO:tensorflow:loss = 0.009596225, step = 37301 (0.190 sec)\n",
            "INFO:tensorflow:global_step/sec: 529.855\n",
            "INFO:tensorflow:loss = 0.0058265403, step = 37401 (0.192 sec)\n",
            "INFO:tensorflow:global_step/sec: 528.3\n",
            "INFO:tensorflow:loss = 0.009873065, step = 37501 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 559.795\n",
            "INFO:tensorflow:loss = 0.00879278, step = 37601 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 558.522\n",
            "INFO:tensorflow:loss = 0.011074197, step = 37701 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.019\n",
            "INFO:tensorflow:loss = 0.012909027, step = 37801 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 547.765\n",
            "INFO:tensorflow:loss = 0.010248518, step = 37901 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 549.995\n",
            "INFO:tensorflow:loss = 0.01310011, step = 38001 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 536.006\n",
            "INFO:tensorflow:loss = 0.010627099, step = 38101 (0.186 sec)\n",
            "INFO:tensorflow:global_step/sec: 560.579\n",
            "INFO:tensorflow:loss = 0.016164748, step = 38201 (0.178 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.488\n",
            "INFO:tensorflow:loss = 0.01696628, step = 38301 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 553.097\n",
            "INFO:tensorflow:loss = 0.01689392, step = 38401 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 550.952\n",
            "INFO:tensorflow:loss = 0.009807292, step = 38501 (0.182 sec)\n",
            "INFO:tensorflow:global_step/sec: 545.268\n",
            "INFO:tensorflow:loss = 0.011835812, step = 38601 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 556.047\n",
            "INFO:tensorflow:loss = 0.0080721695, step = 38701 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.638\n",
            "INFO:tensorflow:loss = 0.008728029, step = 38801 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.833\n",
            "INFO:tensorflow:loss = 0.014676143, step = 38901 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 551.095\n",
            "INFO:tensorflow:loss = 0.00951361, step = 39001 (0.181 sec)\n",
            "INFO:tensorflow:global_step/sec: 542.388\n",
            "INFO:tensorflow:loss = 0.012281878, step = 39101 (0.185 sec)\n",
            "INFO:tensorflow:global_step/sec: 531.816\n",
            "INFO:tensorflow:loss = 0.008037869, step = 39201 (0.188 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.295\n",
            "INFO:tensorflow:loss = 0.0076594995, step = 39301 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 548.636\n",
            "INFO:tensorflow:loss = 0.007676459, step = 39401 (0.183 sec)\n",
            "INFO:tensorflow:global_step/sec: 554.115\n",
            "INFO:tensorflow:loss = 0.006689015, step = 39501 (0.180 sec)\n",
            "INFO:tensorflow:global_step/sec: 557.908\n",
            "INFO:tensorflow:loss = 0.010154917, step = 39601 (0.179 sec)\n",
            "INFO:tensorflow:global_step/sec: 541.959\n",
            "INFO:tensorflow:loss = 0.008096306, step = 39701 (0.184 sec)\n",
            "INFO:tensorflow:global_step/sec: 563.428\n",
            "INFO:tensorflow:loss = 0.012098092, step = 39801 (0.177 sec)\n",
            "INFO:tensorflow:global_step/sec: 552.3\n",
            "INFO:tensorflow:loss = 0.01609891, step = 39901 (0.181 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 40000 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights_with_complexity_regularization/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:05:10\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-40000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't0_1_layer_dnn' dict for global step 40000: architecture/adanet/ensembles = \n",
            "a\n",
            "\u003eadanet/iteration_0/ensemble_t0_1_layer_dnn/architecture/adanetB\u0015\u0008\u0007\u0012\u0000B\u000f| 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036191724, average_loss/adanet/subnetwork = 0.036273483, average_loss/adanet/uniform_average_ensemble = 0.036273483, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046480708, loss/adanet/subnetwork = 0.046664447, loss/adanet/uniform_average_ensemble = 0.046664447, prediction/mean/adanet/adanet_weighted_ensemble = 3.1387463, prediction/mean/adanet/subnetwork = 3.140079, prediction/mean/adanet/uniform_average_ensemble = 3.140079\n",
            "INFO:tensorflow:Saving candidate 't1_1_layer_dnn' dict for global step 40000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_1_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 1_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035279963, average_loss/adanet/subnetwork = 0.03862485, average_loss/adanet/uniform_average_ensemble = 0.036194302, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.042149186, loss/adanet/subnetwork = 0.05337928, loss/adanet/uniform_average_ensemble = 0.048834026, prediction/mean/adanet/adanet_weighted_ensemble = 3.0863802, prediction/mean/adanet/subnetwork = 3.1667511, prediction/mean/adanet/uniform_average_ensemble = 3.153415\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 40000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035131905, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.043354478, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.0939205, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:05:11\n",
            "INFO:tensorflow:Saving dict for global step 40000: average_loss = 0.035131905, average_loss/adanet/adanet_weighted_ensemble = 0.035131905, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, global_step = 40000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.043354478, loss/adanet/adanet_weighted_ensemble = 0.043354478, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean = 3.0939205, prediction/mean/adanet/adanet_weighted_ensemble = 3.0939205, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 40000: /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-40000\n",
            "INFO:tensorflow:Loss for final step: 0.010708799.\n",
            "INFO:tensorflow:Finished training Adanet iteration 1\n",
            "INFO:tensorflow:Beginning bookkeeping phase for iteration 1\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights_with_complexity_regularization/architecture-0.txt: ['0:1_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 1\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Starting ensemble evaluation for iteration 1\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-40000\n",
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n",
            "INFO:tensorflow:Encountered end of input after 14 evaluations\n",
            "INFO:tensorflow:Computed ensemble metrics: adanet_loss/t0_1_layer_dnn = 0.031534, adanet_loss/t1_1_layer_dnn = 0.027044, adanet_loss/t1_2_layer_dnn = 0.026946\n",
            "INFO:tensorflow:Finished ensemble evaluation for iteration 1\n",
            "INFO:tensorflow:'t1_2_layer_dnn' at index 2 is moving onto the next iteration\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights_with_complexity_regularization/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Warm-starting from: (u'/tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-40000',)\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss/biased; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense_1/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense_1/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t1_2_layer_dnn/adanet_loss; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: global_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss/local_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense_1/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_0/logits/mixture_weight; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t0_1_layer_dnn/adanet_loss; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense_2/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t1_2_layer_dnn/adanet/iteration_1/candidate_t1_2_layer_dnn/adanet_loss/biased; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/logits/mixture_weight; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/subnetwork/dense/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/candidate_t0_1_layer_dnn/adanet_loss; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense_1/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/ensemble_t0_1_layer_dnn/weighted_subnetwork_0/logits/mixture_weight; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t1_2_layer_dnn/adanet/iteration_1/candidate_t1_2_layer_dnn/adanet_loss/local_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t0_1_layer_dnn/adanet/iteration_1/candidate_t0_1_layer_dnn/adanet_loss/biased; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/candidate_t0_1_layer_dnn/adanet/iteration_1/candidate_t0_1_layer_dnn/adanet_loss/local_step; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense_2/bias; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/ensemble_t1_2_layer_dnn/weighted_subnetwork_1/subnetwork/dense/kernel; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_1/train_op/is_over/is_over_var_fn/is_over_var; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Warm-starting variable: adanet/iteration_0/train_op/is_over/is_over_var_fn/is_over_var; prev_var_name: Unchanged\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Overwriting checkpoint with new graph for iteration 2 to /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-40000\n",
            "WARNING:tensorflow:`tf.train.start_queue_runners()` was called when no queue runners were defined. You can safely remove the call to this deprecated function.\n",
            "INFO:tensorflow:Finished bookkeeping phase for iteration 1\n",
            "INFO:tensorflow:Beginning training AdaNet iteration 2\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights_with_complexity_regularization/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Create CheckpointSaverHook.\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/increment.ckpt-2\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving checkpoints for 40000 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:loss = 0.011462239, step = 40001\n",
            "INFO:tensorflow:global_step/sec: 104.456\n",
            "INFO:tensorflow:loss = 0.01609045, step = 40101 (0.958 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.939\n",
            "INFO:tensorflow:loss = 0.014762738, step = 40201 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 500.448\n",
            "INFO:tensorflow:loss = 0.010399132, step = 40301 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 510.637\n",
            "INFO:tensorflow:loss = 0.0130894855, step = 40401 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 493.864\n",
            "INFO:tensorflow:loss = 0.0075515686, step = 40501 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 495.298\n",
            "INFO:tensorflow:loss = 0.00787143, step = 40601 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 491.062\n",
            "INFO:tensorflow:loss = 0.00785598, step = 40701 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 493.771\n",
            "INFO:tensorflow:loss = 0.01000301, step = 40801 (0.203 sec)\n",
            "INFO:tensorflow:global_step/sec: 506.337\n",
            "INFO:tensorflow:loss = 0.011615329, step = 40901 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 497.629\n",
            "INFO:tensorflow:loss = 0.015532652, step = 41001 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 500.576\n",
            "INFO:tensorflow:loss = 0.011624692, step = 41101 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 496.648\n",
            "INFO:tensorflow:loss = 0.014772133, step = 41201 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.026\n",
            "INFO:tensorflow:loss = 0.011835646, step = 41301 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 515.188\n",
            "INFO:tensorflow:loss = 0.0057152715, step = 41401 (0.194 sec)\n",
            "INFO:tensorflow:global_step/sec: 509.58\n",
            "INFO:tensorflow:loss = 0.00609661, step = 41501 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 510.402\n",
            "INFO:tensorflow:loss = 0.0045516635, step = 41601 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 503.671\n",
            "INFO:tensorflow:loss = 0.0111754, step = 41701 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 509.824\n",
            "INFO:tensorflow:loss = 0.0098748375, step = 41801 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 373.059\n",
            "INFO:tensorflow:loss = 0.009806033, step = 41901 (0.268 sec)\n",
            "INFO:tensorflow:global_step/sec: 478.984\n",
            "INFO:tensorflow:loss = 0.0074519976, step = 42001 (0.208 sec)\n",
            "INFO:tensorflow:global_step/sec: 509.038\n",
            "INFO:tensorflow:loss = 0.017870229, step = 42101 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 507.331\n",
            "INFO:tensorflow:loss = 0.010496886, step = 42201 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 488.024\n",
            "INFO:tensorflow:loss = 0.0077026156, step = 42301 (0.205 sec)\n",
            "INFO:tensorflow:global_step/sec: 501.105\n",
            "INFO:tensorflow:loss = 0.010206224, step = 42401 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.131\n",
            "INFO:tensorflow:loss = 0.015199267, step = 42501 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 508.849\n",
            "INFO:tensorflow:loss = 0.0055285767, step = 42601 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 500.41\n",
            "INFO:tensorflow:loss = 0.013261177, step = 42701 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 495.803\n",
            "INFO:tensorflow:loss = 0.008539056, step = 42801 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 496.303\n",
            "INFO:tensorflow:loss = 0.006885643, step = 42901 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 453.385\n",
            "INFO:tensorflow:loss = 0.011635002, step = 43001 (0.218 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.333\n",
            "INFO:tensorflow:loss = 0.006786813, step = 43101 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.677\n",
            "INFO:tensorflow:loss = 0.011771876, step = 43201 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 506.637\n",
            "INFO:tensorflow:loss = 0.019987954, step = 43301 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 509.116\n",
            "INFO:tensorflow:loss = 0.01240884, step = 43401 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 500.252\n",
            "INFO:tensorflow:loss = 0.021238232, step = 43501 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 501.015\n",
            "INFO:tensorflow:loss = 0.009588818, step = 43601 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 497.067\n",
            "INFO:tensorflow:loss = 0.010194117, step = 43701 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 498.502\n",
            "INFO:tensorflow:loss = 0.008759463, step = 43801 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 496.199\n",
            "INFO:tensorflow:loss = 0.006166561, step = 43901 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.858\n",
            "INFO:tensorflow:loss = 0.011860265, step = 44001 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 498.676\n",
            "INFO:tensorflow:loss = 0.0113264285, step = 44101 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 491.053\n",
            "INFO:tensorflow:loss = 0.010237283, step = 44201 (0.203 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.757\n",
            "INFO:tensorflow:loss = 0.005525785, step = 44301 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.96\n",
            "INFO:tensorflow:loss = 0.009989336, step = 44401 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 342.368\n",
            "INFO:tensorflow:loss = 0.009202846, step = 44501 (0.292 sec)\n",
            "INFO:tensorflow:global_step/sec: 510.764\n",
            "INFO:tensorflow:loss = 0.007731162, step = 44601 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.311\n",
            "INFO:tensorflow:loss = 0.008330265, step = 44701 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 456.363\n",
            "INFO:tensorflow:loss = 0.0067990944, step = 44801 (0.219 sec)\n",
            "INFO:tensorflow:global_step/sec: 496.428\n",
            "INFO:tensorflow:loss = 0.0072816894, step = 44901 (0.201 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 45000 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights_with_complexity_regularization/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:05:43\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-45000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 45000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035131905, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.043354478, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.0939205, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving candidate 't2_2_layer_dnn' dict for global step 45000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_2_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03665525, average_loss/adanet/subnetwork = 0.047327004, average_loss/adanet/uniform_average_ensemble = 0.038430035, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.046183225, loss/adanet/subnetwork = 0.065781765, loss/adanet/uniform_average_ensemble = 0.05305232, prediction/mean/adanet/adanet_weighted_ensemble = 3.084328, prediction/mean/adanet/subnetwork = 3.0880663, prediction/mean/adanet/uniform_average_ensemble = 3.124987\n",
            "INFO:tensorflow:Saving candidate 't2_3_layer_dnn' dict for global step 45000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_3_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 3_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03598025, average_loss/adanet/subnetwork = 0.044495832, average_loss/adanet/uniform_average_ensemble = 0.037772827, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.044780396, loss/adanet/subnetwork = 0.059036292, loss/adanet/uniform_average_ensemble = 0.051192496, prediction/mean/adanet/adanet_weighted_ensemble = 3.0848663, prediction/mean/adanet/subnetwork = 3.098168, prediction/mean/adanet/uniform_average_ensemble = 3.1283538\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:05:46\n",
            "INFO:tensorflow:Saving dict for global step 45000: average_loss = 0.035131905, average_loss/adanet/adanet_weighted_ensemble = 0.035131905, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, global_step = 45000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.043354478, loss/adanet/adanet_weighted_ensemble = 0.043354478, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean = 3.0939205, prediction/mean/adanet/adanet_weighted_ensemble = 3.0939205, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 45000: /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-45000\n",
            "INFO:tensorflow:global_step/sec: 13.3009\n",
            "INFO:tensorflow:loss = 0.0072410866, step = 45001 (7.519 sec)\n",
            "INFO:tensorflow:global_step/sec: 496.823\n",
            "INFO:tensorflow:loss = 0.0062870784, step = 45101 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 513.379\n",
            "INFO:tensorflow:loss = 0.008622237, step = 45201 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 484.231\n",
            "INFO:tensorflow:loss = 0.016372139, step = 45301 (0.206 sec)\n",
            "INFO:tensorflow:global_step/sec: 490.302\n",
            "INFO:tensorflow:loss = 0.0052740234, step = 45401 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 497.824\n",
            "INFO:tensorflow:loss = 0.0066589466, step = 45501 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 501.11\n",
            "INFO:tensorflow:loss = 0.013444075, step = 45601 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 506.401\n",
            "INFO:tensorflow:loss = 0.008013896, step = 45701 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 499.715\n",
            "INFO:tensorflow:loss = 0.01245831, step = 45801 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 360.746\n",
            "INFO:tensorflow:loss = 0.00811184, step = 45901 (0.277 sec)\n",
            "INFO:tensorflow:global_step/sec: 367.423\n",
            "INFO:tensorflow:loss = 0.009929679, step = 46001 (0.271 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.691\n",
            "INFO:tensorflow:loss = 0.012348412, step = 46101 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 490.309\n",
            "INFO:tensorflow:loss = 0.012803269, step = 46201 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 495.496\n",
            "INFO:tensorflow:loss = 0.006241231, step = 46301 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 496.238\n",
            "INFO:tensorflow:loss = 0.0114059085, step = 46401 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 495.847\n",
            "INFO:tensorflow:loss = 0.00980523, step = 46501 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 477.441\n",
            "INFO:tensorflow:loss = 0.0066221943, step = 46601 (0.210 sec)\n",
            "INFO:tensorflow:global_step/sec: 503.309\n",
            "INFO:tensorflow:loss = 0.009447148, step = 46701 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.459\n",
            "INFO:tensorflow:loss = 0.009256136, step = 46801 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 498.91\n",
            "INFO:tensorflow:loss = 0.0050060377, step = 46901 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 498.619\n",
            "INFO:tensorflow:loss = 0.0061322637, step = 47001 (0.203 sec)\n",
            "INFO:tensorflow:global_step/sec: 492.339\n",
            "INFO:tensorflow:loss = 0.007152724, step = 47101 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 500.498\n",
            "INFO:tensorflow:loss = 0.008765616, step = 47201 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 480.857\n",
            "INFO:tensorflow:loss = 0.009734501, step = 47301 (0.205 sec)\n",
            "INFO:tensorflow:global_step/sec: 510.569\n",
            "INFO:tensorflow:loss = 0.010123188, step = 47401 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 496.243\n",
            "INFO:tensorflow:loss = 0.0056881583, step = 47501 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 499.845\n",
            "INFO:tensorflow:loss = 0.008157105, step = 47601 (0.203 sec)\n",
            "INFO:tensorflow:global_step/sec: 497.537\n",
            "INFO:tensorflow:loss = 0.01580382, step = 47701 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 507.421\n",
            "INFO:tensorflow:loss = 0.01510094, step = 47801 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.101\n",
            "INFO:tensorflow:loss = 0.0061331643, step = 47901 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 498.348\n",
            "INFO:tensorflow:loss = 0.013449804, step = 48001 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 499.328\n",
            "INFO:tensorflow:loss = 0.014141675, step = 48101 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.048\n",
            "INFO:tensorflow:loss = 0.014949798, step = 48201 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 501.547\n",
            "INFO:tensorflow:loss = 0.012498222, step = 48301 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 492.861\n",
            "INFO:tensorflow:loss = 0.007322088, step = 48401 (0.203 sec)\n",
            "INFO:tensorflow:global_step/sec: 347.423\n",
            "INFO:tensorflow:loss = 0.007297946, step = 48501 (0.288 sec)\n",
            "INFO:tensorflow:global_step/sec: 488.811\n",
            "INFO:tensorflow:loss = 0.010770537, step = 48601 (0.205 sec)\n",
            "INFO:tensorflow:global_step/sec: 495.624\n",
            "INFO:tensorflow:loss = 0.011933723, step = 48701 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 503.365\n",
            "INFO:tensorflow:loss = 0.012660578, step = 48801 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 392.243\n",
            "INFO:tensorflow:loss = 0.01893247, step = 48901 (0.260 sec)\n",
            "INFO:tensorflow:global_step/sec: 345.417\n",
            "INFO:tensorflow:loss = 0.018313702, step = 49001 (0.289 sec)\n",
            "INFO:tensorflow:global_step/sec: 421.147\n",
            "INFO:tensorflow:loss = 0.011175569, step = 49101 (0.232 sec)\n",
            "INFO:tensorflow:global_step/sec: 496.044\n",
            "INFO:tensorflow:loss = 0.01238155, step = 49201 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 497.869\n",
            "INFO:tensorflow:loss = 0.0047719865, step = 49301 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.53\n",
            "INFO:tensorflow:loss = 0.008926766, step = 49401 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 506.801\n",
            "INFO:tensorflow:loss = 0.013850714, step = 49501 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 489.004\n",
            "INFO:tensorflow:loss = 0.008156413, step = 49601 (0.205 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.313\n",
            "INFO:tensorflow:loss = 0.007694022, step = 49701 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.116\n",
            "INFO:tensorflow:loss = 0.009452383, step = 49801 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 503.111\n",
            "INFO:tensorflow:loss = 0.014773121, step = 49901 (0.199 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 50000 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights_with_complexity_regularization/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:06:01\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-50000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 50000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035131905, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.043354478, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.0939205, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving candidate 't2_2_layer_dnn' dict for global step 50000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_2_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036650807, average_loss/adanet/subnetwork = 0.056401946, average_loss/adanet/uniform_average_ensemble = 0.037080817, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.04455299, loss/adanet/subnetwork = 0.06400509, loss/adanet/uniform_average_ensemble = 0.047858924, prediction/mean/adanet/adanet_weighted_ensemble = 3.072265, prediction/mean/adanet/subnetwork = 2.9943178, prediction/mean/adanet/uniform_average_ensemble = 3.0937374\n",
            "INFO:tensorflow:Saving candidate 't2_3_layer_dnn' dict for global step 50000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_3_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 3_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03629343, average_loss/adanet/subnetwork = 0.06374278, average_loss/adanet/uniform_average_ensemble = 0.03804928, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.04418526, loss/adanet/subnetwork = 0.07257131, loss/adanet/uniform_average_ensemble = 0.04967241, prediction/mean/adanet/adanet_weighted_ensemble = 3.075336, prediction/mean/adanet/subnetwork = 2.9759707, prediction/mean/adanet/uniform_average_ensemble = 3.0876217\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:06:02\n",
            "INFO:tensorflow:Saving dict for global step 50000: average_loss = 0.036650807, average_loss/adanet/adanet_weighted_ensemble = 0.036650807, average_loss/adanet/subnetwork = 0.056401946, average_loss/adanet/uniform_average_ensemble = 0.037080817, global_step = 50000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.04455299, loss/adanet/adanet_weighted_ensemble = 0.04455299, loss/adanet/subnetwork = 0.06400509, loss/adanet/uniform_average_ensemble = 0.047858924, prediction/mean = 3.072265, prediction/mean/adanet/adanet_weighted_ensemble = 3.072265, prediction/mean/adanet/subnetwork = 2.9943178, prediction/mean/adanet/uniform_average_ensemble = 3.0937374\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 50000: /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-50000\n",
            "INFO:tensorflow:global_step/sec: 16.7191\n",
            "INFO:tensorflow:loss = 0.013374895, step = 50001 (5.983 sec)\n",
            "INFO:tensorflow:global_step/sec: 494.893\n",
            "INFO:tensorflow:loss = 0.005973286, step = 50101 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.763\n",
            "INFO:tensorflow:loss = 0.010135171, step = 50201 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 503.294\n",
            "INFO:tensorflow:loss = 0.011500949, step = 50301 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 500.513\n",
            "INFO:tensorflow:loss = 0.008156536, step = 50401 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 500.666\n",
            "INFO:tensorflow:loss = 0.01008632, step = 50501 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 336.889\n",
            "INFO:tensorflow:loss = 0.007357282, step = 50601 (0.297 sec)\n",
            "INFO:tensorflow:global_step/sec: 498.942\n",
            "INFO:tensorflow:loss = 0.006511369, step = 50701 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 492.541\n",
            "INFO:tensorflow:loss = 0.0092080105, step = 50801 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.988\n",
            "INFO:tensorflow:loss = 0.013389524, step = 50901 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 496.007\n",
            "INFO:tensorflow:loss = 0.0060604624, step = 51001 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 461.806\n",
            "INFO:tensorflow:loss = 0.012158271, step = 51101 (0.216 sec)\n",
            "INFO:tensorflow:global_step/sec: 461.444\n",
            "INFO:tensorflow:loss = 0.0048439726, step = 51201 (0.217 sec)\n",
            "INFO:tensorflow:global_step/sec: 445.428\n",
            "INFO:tensorflow:loss = 0.009012477, step = 51301 (0.224 sec)\n",
            "INFO:tensorflow:global_step/sec: 452.178\n",
            "INFO:tensorflow:loss = 0.005870981, step = 51401 (0.221 sec)\n",
            "INFO:tensorflow:global_step/sec: 450.757\n",
            "INFO:tensorflow:loss = 0.015410252, step = 51501 (0.222 sec)\n",
            "INFO:tensorflow:global_step/sec: 441.23\n",
            "INFO:tensorflow:loss = 0.0067593455, step = 51601 (0.226 sec)\n",
            "INFO:tensorflow:global_step/sec: 430.209\n",
            "INFO:tensorflow:loss = 0.011005949, step = 51701 (0.232 sec)\n",
            "INFO:tensorflow:global_step/sec: 452.501\n",
            "INFO:tensorflow:loss = 0.007561092, step = 51801 (0.221 sec)\n",
            "INFO:tensorflow:global_step/sec: 454.005\n",
            "INFO:tensorflow:loss = 0.009498428, step = 51901 (0.220 sec)\n",
            "INFO:tensorflow:global_step/sec: 447.996\n",
            "INFO:tensorflow:loss = 0.008596795, step = 52001 (0.223 sec)\n",
            "INFO:tensorflow:global_step/sec: 451.52\n",
            "INFO:tensorflow:loss = 0.007816934, step = 52101 (0.221 sec)\n",
            "INFO:tensorflow:global_step/sec: 459.447\n",
            "INFO:tensorflow:loss = 0.0048704185, step = 52201 (0.218 sec)\n",
            "INFO:tensorflow:global_step/sec: 446.485\n",
            "INFO:tensorflow:loss = 0.008411579, step = 52301 (0.224 sec)\n",
            "INFO:tensorflow:global_step/sec: 423.082\n",
            "INFO:tensorflow:loss = 0.006491867, step = 52401 (0.236 sec)\n",
            "INFO:tensorflow:global_step/sec: 450.452\n",
            "INFO:tensorflow:loss = 0.010266993, step = 52501 (0.222 sec)\n",
            "INFO:tensorflow:global_step/sec: 435.867\n",
            "INFO:tensorflow:loss = 0.008442799, step = 52601 (0.230 sec)\n",
            "INFO:tensorflow:global_step/sec: 448.354\n",
            "INFO:tensorflow:loss = 0.0135071175, step = 52701 (0.223 sec)\n",
            "INFO:tensorflow:global_step/sec: 439.078\n",
            "INFO:tensorflow:loss = 0.010923101, step = 52801 (0.228 sec)\n",
            "INFO:tensorflow:global_step/sec: 307.471\n",
            "INFO:tensorflow:loss = 0.010644123, step = 52901 (0.326 sec)\n",
            "INFO:tensorflow:global_step/sec: 460.929\n",
            "INFO:tensorflow:loss = 0.014881212, step = 53001 (0.216 sec)\n",
            "INFO:tensorflow:global_step/sec: 455.678\n",
            "INFO:tensorflow:loss = 0.011817278, step = 53101 (0.219 sec)\n",
            "INFO:tensorflow:global_step/sec: 460.191\n",
            "INFO:tensorflow:loss = 0.007929478, step = 53201 (0.217 sec)\n",
            "INFO:tensorflow:global_step/sec: 441.156\n",
            "INFO:tensorflow:loss = 0.0063758497, step = 53301 (0.227 sec)\n",
            "INFO:tensorflow:global_step/sec: 435.127\n",
            "INFO:tensorflow:loss = 0.009661921, step = 53401 (0.230 sec)\n",
            "INFO:tensorflow:global_step/sec: 446.592\n",
            "INFO:tensorflow:loss = 0.009159315, step = 53501 (0.224 sec)\n",
            "INFO:tensorflow:global_step/sec: 472.706\n",
            "INFO:tensorflow:loss = 0.010908134, step = 53601 (0.211 sec)\n",
            "INFO:tensorflow:global_step/sec: 513.973\n",
            "INFO:tensorflow:loss = 0.005155122, step = 53701 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.058\n",
            "INFO:tensorflow:loss = 0.012908818, step = 53801 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.093\n",
            "INFO:tensorflow:loss = 0.0090850815, step = 53901 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 513.128\n",
            "INFO:tensorflow:loss = 0.0044321474, step = 54001 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 499.418\n",
            "INFO:tensorflow:loss = 0.01067212, step = 54101 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 498.736\n",
            "INFO:tensorflow:loss = 0.011219837, step = 54201 (0.203 sec)\n",
            "INFO:tensorflow:global_step/sec: 485.263\n",
            "INFO:tensorflow:loss = 0.009063743, step = 54301 (0.203 sec)\n",
            "INFO:tensorflow:global_step/sec: 495.042\n",
            "INFO:tensorflow:loss = 0.0068207923, step = 54401 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 506.273\n",
            "INFO:tensorflow:loss = 0.0075606448, step = 54501 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 500.591\n",
            "INFO:tensorflow:loss = 0.008071984, step = 54601 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.459\n",
            "INFO:tensorflow:loss = 0.0114433365, step = 54701 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 500.753\n",
            "INFO:tensorflow:loss = 0.015939377, step = 54801 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 508.107\n",
            "INFO:tensorflow:loss = 0.009774547, step = 54901 (0.197 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 55000 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights_with_complexity_regularization/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:06:17\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-55000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 55000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035131905, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.043354478, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.0939205, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving candidate 't2_2_layer_dnn' dict for global step 55000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_2_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.03693859, average_loss/adanet/subnetwork = 0.04715966, average_loss/adanet/uniform_average_ensemble = 0.03509828, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.0439814, loss/adanet/subnetwork = 0.052383907, loss/adanet/uniform_average_ensemble = 0.044210464, prediction/mean/adanet/adanet_weighted_ensemble = 3.0647383, prediction/mean/adanet/subnetwork = 3.014184, prediction/mean/adanet/uniform_average_ensemble = 3.1003594\n",
            "INFO:tensorflow:Saving candidate 't2_3_layer_dnn' dict for global step 55000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_3_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 3_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.036713652, average_loss/adanet/subnetwork = 0.04596351, average_loss/adanet/uniform_average_ensemble = 0.03755821, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.044045992, loss/adanet/subnetwork = 0.05456455, loss/adanet/uniform_average_ensemble = 0.048664063, prediction/mean/adanet/adanet_weighted_ensemble = 3.0690026, prediction/mean/adanet/subnetwork = 3.0800374, prediction/mean/adanet/uniform_average_ensemble = 3.1223106\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:06:19\n",
            "INFO:tensorflow:Saving dict for global step 55000: average_loss = 0.03693859, average_loss/adanet/adanet_weighted_ensemble = 0.03693859, average_loss/adanet/subnetwork = 0.04715966, average_loss/adanet/uniform_average_ensemble = 0.03509828, global_step = 55000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.0439814, loss/adanet/adanet_weighted_ensemble = 0.0439814, loss/adanet/subnetwork = 0.052383907, loss/adanet/uniform_average_ensemble = 0.044210464, prediction/mean = 3.0647383, prediction/mean/adanet/adanet_weighted_ensemble = 3.0647383, prediction/mean/adanet/subnetwork = 3.014184, prediction/mean/adanet/uniform_average_ensemble = 3.1003594\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 55000: /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-55000\n",
            "INFO:tensorflow:global_step/sec: 16.0627\n",
            "INFO:tensorflow:loss = 0.01599465, step = 55001 (6.227 sec)\n",
            "INFO:tensorflow:global_step/sec: 496.91\n",
            "INFO:tensorflow:loss = 0.008851289, step = 55101 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 503.031\n",
            "INFO:tensorflow:loss = 0.010140075, step = 55201 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.745\n",
            "INFO:tensorflow:loss = 0.01458632, step = 55301 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 504.185\n",
            "INFO:tensorflow:loss = 0.008713124, step = 55401 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 490.554\n",
            "INFO:tensorflow:loss = 0.011734816, step = 55501 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 508.179\n",
            "INFO:tensorflow:loss = 0.010897214, step = 55601 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 512.492\n",
            "INFO:tensorflow:loss = 0.0074819694, step = 55701 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 505.229\n",
            "INFO:tensorflow:loss = 0.013045141, step = 55801 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 512.329\n",
            "INFO:tensorflow:loss = 0.008449868, step = 55901 (0.195 sec)\n",
            "INFO:tensorflow:global_step/sec: 507.241\n",
            "INFO:tensorflow:loss = 0.006965699, step = 56001 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 506.935\n",
            "INFO:tensorflow:loss = 0.0077072345, step = 56101 (0.197 sec)\n",
            "INFO:tensorflow:global_step/sec: 506.847\n",
            "INFO:tensorflow:loss = 0.006649466, step = 56201 (0.198 sec)\n",
            "INFO:tensorflow:global_step/sec: 499.775\n",
            "INFO:tensorflow:loss = 0.01002968, step = 56301 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.662\n",
            "INFO:tensorflow:loss = 0.0062373728, step = 56401 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 500.633\n",
            "INFO:tensorflow:loss = 0.0058091306, step = 56501 (0.200 sec)\n",
            "INFO:tensorflow:global_step/sec: 495.212\n",
            "INFO:tensorflow:loss = 0.007877046, step = 56601 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 488.226\n",
            "INFO:tensorflow:loss = 0.011366559, step = 56701 (0.205 sec)\n",
            "INFO:tensorflow:global_step/sec: 494.425\n",
            "INFO:tensorflow:loss = 0.008756493, step = 56801 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.474\n",
            "INFO:tensorflow:loss = 0.0053733205, step = 56901 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 484.743\n",
            "INFO:tensorflow:loss = 0.012768038, step = 57001 (0.206 sec)\n",
            "INFO:tensorflow:global_step/sec: 497.901\n",
            "INFO:tensorflow:loss = 0.012915339, step = 57101 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 492.387\n",
            "INFO:tensorflow:loss = 0.0076998565, step = 57201 (0.203 sec)\n",
            "INFO:tensorflow:global_step/sec: 487.405\n",
            "INFO:tensorflow:loss = 0.011124671, step = 57301 (0.205 sec)\n",
            "INFO:tensorflow:global_step/sec: 509.884\n",
            "INFO:tensorflow:loss = 0.00619088, step = 57401 (0.196 sec)\n",
            "INFO:tensorflow:global_step/sec: 349.784\n",
            "INFO:tensorflow:loss = 0.010256084, step = 57501 (0.297 sec)\n",
            "INFO:tensorflow:global_step/sec: 456.094\n",
            "INFO:tensorflow:loss = 0.0076283608, step = 57601 (0.208 sec)\n",
            "INFO:tensorflow:global_step/sec: 447.267\n",
            "INFO:tensorflow:loss = 0.008633638, step = 57701 (0.229 sec)\n",
            "INFO:tensorflow:global_step/sec: 403.492\n",
            "INFO:tensorflow:loss = 0.010228822, step = 57801 (0.243 sec)\n",
            "INFO:tensorflow:global_step/sec: 402.539\n",
            "INFO:tensorflow:loss = 0.008503232, step = 57901 (0.249 sec)\n",
            "INFO:tensorflow:global_step/sec: 410.502\n",
            "INFO:tensorflow:loss = 0.01286545, step = 58001 (0.244 sec)\n",
            "INFO:tensorflow:global_step/sec: 498.621\n",
            "INFO:tensorflow:loss = 0.011636835, step = 58101 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 482.838\n",
            "INFO:tensorflow:loss = 0.013420696, step = 58201 (0.207 sec)\n",
            "INFO:tensorflow:global_step/sec: 484.66\n",
            "INFO:tensorflow:loss = 0.017234843, step = 58301 (0.206 sec)\n",
            "INFO:tensorflow:global_step/sec: 481.228\n",
            "INFO:tensorflow:loss = 0.01596462, step = 58401 (0.207 sec)\n",
            "INFO:tensorflow:global_step/sec: 502.389\n",
            "INFO:tensorflow:loss = 0.0077269534, step = 58501 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 494.694\n",
            "INFO:tensorflow:loss = 0.010517777, step = 58601 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 492.703\n",
            "INFO:tensorflow:loss = 0.006440703, step = 58701 (0.203 sec)\n",
            "INFO:tensorflow:global_step/sec: 497.193\n",
            "INFO:tensorflow:loss = 0.010009007, step = 58801 (0.202 sec)\n",
            "INFO:tensorflow:global_step/sec: 488.443\n",
            "INFO:tensorflow:loss = 0.01458048, step = 58901 (0.205 sec)\n",
            "INFO:tensorflow:global_step/sec: 491.77\n",
            "INFO:tensorflow:loss = 0.006805434, step = 59001 (0.206 sec)\n",
            "INFO:tensorflow:global_step/sec: 485.093\n",
            "INFO:tensorflow:loss = 0.009897639, step = 59101 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 488.945\n",
            "INFO:tensorflow:loss = 0.006171802, step = 59201 (0.204 sec)\n",
            "INFO:tensorflow:global_step/sec: 497.354\n",
            "INFO:tensorflow:loss = 0.007347608, step = 59301 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 478.856\n",
            "INFO:tensorflow:loss = 0.006735719, step = 59401 (0.209 sec)\n",
            "INFO:tensorflow:global_step/sec: 481.988\n",
            "INFO:tensorflow:loss = 0.005441291, step = 59501 (0.208 sec)\n",
            "INFO:tensorflow:global_step/sec: 501.673\n",
            "INFO:tensorflow:loss = 0.0072499244, step = 59601 (0.199 sec)\n",
            "INFO:tensorflow:global_step/sec: 497.32\n",
            "INFO:tensorflow:loss = 0.0067425165, step = 59701 (0.201 sec)\n",
            "INFO:tensorflow:global_step/sec: 481.512\n",
            "INFO:tensorflow:loss = 0.009846831, step = 59801 (0.207 sec)\n",
            "INFO:tensorflow:global_step/sec: 438.127\n",
            "INFO:tensorflow:loss = 0.0144964205, step = 59901 (0.237 sec)\n",
            "INFO:tensorflow:Saving checkpoints for 60000 into /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt.\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Importing architecture from /tmp/models/learn_mixture_weights_with_complexity_regularization/architecture-1.txt: ['0:1_layer_dnn', '1:2_layer_dnn'].\n",
            "INFO:tensorflow:Rebuilding iteration 0\n",
            "INFO:tensorflow:Building subnetwork '1_layer_dnn'\n",
            "INFO:tensorflow:Rebuilding iteration 1\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building iteration 2\n",
            "INFO:tensorflow:Building subnetwork '2_layer_dnn'\n",
            "INFO:tensorflow:Building subnetwork '3_layer_dnn'\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2018-12-19-16:06:34\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-60000\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Saving candidate 't1_2_layer_dnn' dict for global step 60000: architecture/adanet/ensembles = \n",
            "o\n",
            "\u003eadanet/iteration_1/ensemble_t1_2_layer_dnn/architecture/adanetB#\u0008\u0007\u0012\u0000B\u001d| 1_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035131905, average_loss/adanet/subnetwork = 0.04038965, average_loss/adanet/uniform_average_ensemble = 0.036551207, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.043354478, loss/adanet/subnetwork = 0.055296417, loss/adanet/uniform_average_ensemble = 0.049044557, prediction/mean/adanet/adanet_weighted_ensemble = 3.0939205, prediction/mean/adanet/subnetwork = 3.1468153, prediction/mean/adanet/uniform_average_ensemble = 3.143447\n",
            "INFO:tensorflow:Saving candidate 't2_2_layer_dnn' dict for global step 60000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_2_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 2_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.035705358, average_loss/adanet/subnetwork = 0.036563545, average_loss/adanet/uniform_average_ensemble = 0.03492362, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.043283053, loss/adanet/subnetwork = 0.049429692, loss/adanet/uniform_average_ensemble = 0.0469208, prediction/mean/adanet/adanet_weighted_ensemble = 3.0724835, prediction/mean/adanet/subnetwork = 3.1345265, prediction/mean/adanet/uniform_average_ensemble = 3.1404736\n",
            "INFO:tensorflow:Saving candidate 't2_3_layer_dnn' dict for global step 60000: architecture/adanet/ensembles = \n",
            "}\n",
            "\u003eadanet/iteration_2/ensemble_t2_3_layer_dnn/architecture/adanetB1\u0008\u0007\u0012\u0000B+| 1_layer_dnn | 2_layer_dnn | 3_layer_dnn |J\u0008\n",
            "\u0006\n",
            "\u0004text, average_loss/adanet/adanet_weighted_ensemble = 0.037045587, average_loss/adanet/subnetwork = 0.047027905, average_loss/adanet/uniform_average_ensemble = 0.038446438, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss/adanet/adanet_weighted_ensemble = 0.044373743, loss/adanet/subnetwork = 0.0658215, loss/adanet/uniform_average_ensemble = 0.05310804, prediction/mean/adanet/adanet_weighted_ensemble = 3.0665076, prediction/mean/adanet/subnetwork = 3.1232636, prediction/mean/adanet/uniform_average_ensemble = 3.1367195\n",
            "INFO:tensorflow:Finished evaluation at 2018-12-19-16:06:35\n",
            "INFO:tensorflow:Saving dict for global step 60000: average_loss = 0.035705358, average_loss/adanet/adanet_weighted_ensemble = 0.035705358, average_loss/adanet/subnetwork = 0.036563545, average_loss/adanet/uniform_average_ensemble = 0.03492362, global_step = 60000, label/mean = 3.1049454, label/mean/adanet/adanet_weighted_ensemble = 3.1049454, label/mean/adanet/subnetwork = 3.1049454, label/mean/adanet/uniform_average_ensemble = 3.1049454, loss = 0.043283053, loss/adanet/adanet_weighted_ensemble = 0.043283053, loss/adanet/subnetwork = 0.049429692, loss/adanet/uniform_average_ensemble = 0.0469208, prediction/mean = 3.0724835, prediction/mean/adanet/adanet_weighted_ensemble = 3.0724835, prediction/mean/adanet/subnetwork = 3.1345265, prediction/mean/adanet/uniform_average_ensemble = 3.1404736\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 60000: /tmp/models/learn_mixture_weights_with_complexity_regularization/model.ckpt-60000\n",
            "INFO:tensorflow:Loss for final step: 0.006377102.\n",
            "INFO:tensorflow:Finished training Adanet iteration 2\n",
            "Loss: 0.035705358\n",
            "Uniform average loss: 0.03492362\n",
            "Architecture: | 1_layer_dnn | 2_layer_dnn | 2_layer_dnn |\n"
          ]
        }
      ],
      "source": [
        "#@test {\"skip\": true}\n",
        "results, _ = train_and_evaluate(\"learn_mixture_weights_with_complexity_regularization\", learn_mixture_weights=True, adanet_lambda=.015)\n",
        "print(\"Loss:\", results[\"average_loss\"])\n",
        "print(\"Uniform average loss:\", results[\"average_loss/adanet/uniform_average_ensemble\"])\n",
        "print(\"Architecture:\", ensemble_architecture(results))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "s5RuM582oxo_"
      },
      "source": [
        "Learning the mixture weights with $\\lambda \u003e 0$ produces a model with **0.0354**\n",
        "MSE.\n",
        "\n",
        "Inspecting the ensemble architecture demonstrates the effects of complexity\n",
        "regularization on candidate selection. The selected subnetworks are relatively\n",
        "less complex: unlike in previous runs, the deepest subnetwork has only 2 hidden layers."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ajRhOvHGQu6w"
      },
      "source": [
        "## Conclusion\n",
        "\n",
        "In this tutorial, you were able to explore training an AdaNet model's mixture\n",
        "weights with $\\lambda \\ge 0$. You were also able to compare against building an\n",
        "ensemble formed by always choosing the best candidate subnetwork at each\n",
        "iteration based on it's ability to improve the ensemble's loss on the training\n",
        "set, and averaging their results.\n",
        "\n",
        "Uniform average ensembles work unreasonably well in practice, yet learning the\n",
        "mixture weights with the correct values of $\\lambda$ and $\\beta$ should always\n",
        "produce a better model when candidates have varying complexity. However, this\n",
        "does require some additional hyperparameter tuning, so practically you can train\n",
        "an AdaNet with the default mixture weights and $\\lambda=0$ first, and once you\n",
        "have confirmed that the subnetworks are training correctly, you can tune the\n",
        "mixture weight hyperparameters.\n",
        "\n",
        "While this example explored a regression task, these observations apply to using\n",
        "AdaNet on other tasks like binary-classification and multi-class classification."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "adanet_objective.ipynb",
      "provenance": [],
      "version": "0.3.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
